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In the course of this work, several important issues were examined: hybrid diagnbsticsi knowledge 
engiheerihg costs, user ihterfaces, and the integration of training and job aiding* The term "hybrid 
diagnostics'* refers to the utilization of multiple sources of knowledge in the development of maintenance 
expert systems, in particular (a) dependency modeling (pbtehtially derivable froi engineering databases) and 
(b) heuristic expertise of field technicians. In the area of dependency modeling, one source of knowledge 
identified for the prototype was the test program set of the automatic test station. This information 
provided the specific measurement values arid locatibhs necessary for making measurements during 
troubleshooting. Knowledge ehgiheerihg costs were controlled through use of these test program sets and the 
development of a "glass box" editor which permitted knowledge base modifications during program operation. 

The design for the prdtdtypie ihcdrpdrated human-machine interfaces to promote incremental skill acquisition 
and td mitigate against mental dependence dh the Maihtaiher's Associate. Incremental skill development was 
alsd promoted through end-user interfaces which provide a variety of explanations about the reasoning behind 
the diagnostic prdcess in a given trdubleshddtihg situation. 

In a field demdhstratidh, the prdtdtype Malntainer's Associate received highly favorable ratings for ease 
of use, speed of bperatidh, trbubleshobtihg accuracy* and usefulness for job aiding and trainings This 
project identified additional research, issues in using an expert system diagnostic reasoner as the basis for 
intelligent maintenance training simulations and the need to support mental compilation of test strategies 
from network dependencies with software tools. 
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SUMMARY 



The concept of a Waintairier's Associate calls for a portable, expert- 

system-based job aiding and training device to assist inexperienced electronics 
maintenance technicians. In order to investigate a variety of issues— hybrid 
diagnosis, knowledge enjineering, and user iriterfaces—a prbtbtyj5€5 Maintainer*s 
Assoaate was designed and implemented for troubleshbdtihg portions of the F-IU 
6883 intermediate-level avionics test station. Both system development software 
and delivery software are described. In a field demonstration, the prototype 
system received highly favorable ratings for ease of use, speed of operation, 
troubleshooting accuracy, and usefulness for job aiding and training. Implications 
for future development focused on realizing the training potential of the system, 
enhancing user interfaces, and expanding the problem domain. 
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CHAPTER i: INTROD^K^TON 



T*^^? J^^^<^''Ab an effort conducted by the Denver Research 

Instiiute (DRI) for the Ah Force Human Resources Laboratory. The focus of this 
resesr^h and develCi^mem effort was thie developmerit of the technology 

for a prototype Maintainer's Associate. 

Before this effort is described, the concept of an "associate" for the 

Ql^iy^l^n^'^ce technici^^ from a variety of 

problems, policies, and technologies that have merged together to make its 
realization both desirable and feasible; 



Backgro und 



Current Maintenance Shortcoming s 

There are many widely recognized shortcomings with current job aids^ 
training! and technical documjyitation in electronics maihteharice (Richardson^ 
Keller^ Maxion, Poison, & De3ong, 1985); Technical documentation, for example, 
is paper-based and physically bujkj^ Poor cpordination and insufficient 
cooperation between design engineers and the creators of technical 
documentation have led to problerns of inadequate re and usefulness; the 

information is often out of line with technicians' needs and mental approaches to 

problem-solving. The coordination between technical documentation and 

ir^structional materials used for training, as well as between these materials and 
9?)^®/ '"^sources on the^i^^^ built-in and automatic test equipment^ is also 

Keeping paper-based job aids up to date is another problem, because 
resjponding to and in corpora suggestions from the field are Uhrealistically 
slow. Other current maintenance problems include the need for standardization in 
th^e^acquisition^j^rqcess, th^^^ arid automatic test equipment, the 

demand for more skilled technicians from a less skilled recruit pool, and logistical 
problems throughout maintenance information support systems (Richardson et al., 
i985). 

in addition to these shortcomings, there are a number of trends which 
compound today's rnaintenance task and threaten to make the future of supporting 
weapon systems even more difficult; First, advances in technology have 
complicated rather than simplified maintehahce becausie technology tends to 
increase functionality but not reliability. Second, persohhiel riesources are 
diminishing. IH[ig^hl^^ needed by the military at a time when the 

supply of young persons ol all aptitudes is declining and competition with industry 
for experienced technic^^ cahhot rely on coUriteractihg 

advancing technology's impact on maintenance by recruiting more and brighter 
pers^onnej. A third trend concerns operational requirements of the future. 
Battle scenarios for the late 199^^^^ century call for the ability to 

sustain intense surges; the need for small, highly mobile units; and the capacity to 
mobilize against a more capable threat (Air Force Human Resources Laboratory, 
i9&^h All of these requirements put extra demands on maintenance; 
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. in addition to the current maihtehahce situation, there are two other 

^^*o" *hat have contributed to the cdhcept of a maintainer's associate: (a) 
Department of Defense (DoD) policy^ and (b) technological advanc" 



ices; 



DbD Policy Initiatives 

_ J^°'"®/"ost policies that have contributed to the associate concept 

IS that which pertains to Integrated Diagnostics. This p51icy states that all 
Ule-cycle concerns relevant to maintenance should be considered in an integrated 
fashion; Integrated Diagnostics is a structured process which maximizes the 
effectiveness of diagnostics by integrating pertinent elements such as testability, 
automatic and manual testing, training, maintenance aiding, and technical 
inlormation^ The goal is to minimize equipment failures by addressine 
maintenance and logistics support problems at the beginning of the design phase 
of a system (National Security Industrial Association, 1983, 198^8). 

Another important policy was the DoD logistics R&D initiative to 

replace paper technical orders with an interactive maintenance aiding device 
Rational Security Industrial Association, 198 *a); There are numerous Ongoing 
R&D programs working toward this goal. In the Air Force, the Integrated 
Maintenance Information System (IMISl program Oohnson, 1981) is the first 
program to clearly define the fUrictiorialities of a system to supjport maintenance 
technieians' information needs thrbUgh electronic means. In the Navy, similar 
programs are the Personalized Electrbnic Aid for Maintenance and the Integrated 
?h^^"«5'M "^^^ P^*"^^^ "maintainer's associate" was first used in 

the 1985 Nationai Academy of Sciences Summer Study on Fault Isolation in Air 
hor^e Weapon and Support Systems (Natibnal Academy Preis, in press) to describe 
xf^' r, of the recommendatibris that emerged from this effort was 
that the Air Force should immediately structure a program to develop a 
futule*"^'''^ associate system for a specific application in the near 

TechnologicaiAdvances 

. The concept of an interactive jbb performance aiding and training device 

IS feastbie because of recent advances in artificial intelligence (AI) techniques 
applicable to physical systems, especially in the area of computer programs called 
expert _ systems." Expert systems are able to explain their reasoning, deal with 
uncertainty, and expand to augment their competency; Although there are other 

^^K^So^*?"! *?,,'^^'"-^^ design for testability and maintainability, 

embedded test ("smart" built-in test), off-line test (automatic test program 
generation), and logistics decisibri support, expert systems can be used to address 
the human resources problems bf developing and supporting skilled technical 
personnel through the concept bf a maintainer's associate. 
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Bevelopment of a Maintainer's Associate 



Due to the fact that R&D in the area of a main tainer's associate^ i^^^ 
in the exploratory development stage, there is sometimes confusion between the 
concept of ah associatej the design or plan for AQ_ ^^sociate, and the actual 
prototype device that has been developed and demonstrated. To avoid this 
confusion, the authors cf this report will refer to the concept as it is discussed in 
the following sections; i.e., in terms of the idealized scenario which descrjbes how 
a human technician and a portable machine should act in cooperation to 
troubleshoot maintenance problems. The design or plan for the associate is the 
overall scheme for the prototype which will eventually include features whjch are 
attainable goals, but not all of which were realized in the present effort. The 
name "Maintainer's Associate" will, throughout this report, refer to the actual 
prototype device itself and its features as developed and demonstrated by DRI. 

9 

System Concept 

The concept of an associate system involves a com pater-based device 
with three basic functionss an electronic information resource, a [ob aid, and a 
trainer. Figure 1 illustrates how the system is expected to function as a fle)cible 
integrator of different information resources. The systeni's intern^^ 
should contain a file of engineering design drawings, schematics, illustrated parts 
breakdowns, and basic theories of operation. The system should also interf^^ 
global data supporting the aircraft under repair. The fleet's maintenance history 
and an aircraft's onboard diagnostic and qperatiq^n^al data systems could be 
accessed by data links and made available to the technician through the associate. 
In addition to drawing on this maintenance corporate niemory for the aircraf tj^ the 
associate should also build the memory by acting as an interface to a nianagement 
information system (MIS)^ The technician would use the associate to file reports 
on work in progress and to record new insights of potential use to the technician's 
peers and successors. 

As a job aid, the associate concept involves an expert system capable of 
providirig advice arid directibri for performing diagnostic tasks. The system should 
enable unskilled techniciaris to perform as if they were skilled and to work 
cooperatively with skilled technicians to solve difficult diagnostic problems and 
capture these insights for subsequent use by less experienced technicians. These 
system-user interfaces are expected to elevate the associate above the 
traditional, "cookbook'' job performance aids. They should promote, monitor, and 
use the skill of technicians in a cooperative machine/human system. In contrast 
to current automatic test equipment technology where a fixed sequence of tests is 
followed, the associate is planned to operate froni an exj5ert system knqwje^^ 
base that can develop sequence of tests "on the fly." Thus, the associate concept 
permits the technician to peruse interactively the sp^ce of possible sol^ 
problem. The technician would be able to observe the supporting data on which 
the expert system has based its "solution" and to work in conjunction with the 
computer by assessing its conclusions. 
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INTEGRATED RESOURCES 



• FAUtT DETECTION/FAULT ISOLATION 



PROCEDURES 

• TECHNICAL INFORMATION 

• MAINTENANCE INFORMATION SYSTEM 

• ON-THE-JdB TRAINING 




MIS 
HOST 

COMPUTER 



Update data link 



AIRCRAFT ON 
FLIGHT LINE 



BIT DATA LINK 



aguf&4. Illustration of the Maintainer's Associate System Concept. 
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As a trainer^ the associate should be able to work with the technician 
whose sklH is stiH developing^ Operating in a tutoriai rtibde^ the associate shpllld 
provide troubleshooting practice arid track the technician's progress. the 
technician would be encouraged to anticipate the expert systemVs diagripstic 
feasoriirig. Given its foundation in AI technology, the associate should be able to 
jtastify its decision to its hurriari colleague. The technician should be able to begin 
a tutorial sessibri at cbrivenierit tirhes^ either during downtime in regular 
mainteriarice activities or duririg tirrie spiecifically set aside for training. Since a 
kribwledge base is the sburce bf diagribstic expertise^ the assbciate should be able 
to respond to the techniciari at the appropriate skill level^ based bh a model of the 
user and instructional principles. The model may alsb be tied tb the technician's 
personnel record_ari^d, in the aggregate, to the recbrds bf the entire maintenance 
labor force. Full realization of the maihtainer's assbciate concept would 
integrate the traditionally separate cbricerris of training and job performance 
aiding. 

System Beriefits 

Several beriefits are expected tb result frbm the successful deployment 
of a mairitairier's associate in the field. As digital irifbrrhatibri processing replaces 
the growing volume bf paper technical dbcumeritatibri^ the cUmbersbme bulk of 
paper aids ceases to be a problerri^i In the digital mediurin, irifbrrhatibri is accessed 
faster and riianipulated more easily, \nbther prbjected beriefit of the assbciate is 
the promottori of techriiciari excellence, because the systerri is iriterided tb act as a 
skill multiplier for novice technicians and as a skill integratbr for skilled 
technicians which would capture the corf>brate mernbry of a niairiteriarice corps 
regardless of personnel changes. 

The risks irivblved iri developirig ari assbciate within a 3-yeai^time f rame 
are manageable. This statement is supported by three bbservatibris. Firsts the AI 
technology upon which an assbciate is based has beeri develbpirig thrbUgh R&D for 
over a decade. Secorid, dembnstratibri prototypes have been develbped fbr 
nontrivial systems. Third, the AI software rieeded for an assbciate has appeared 
iri the private sector, iridicating that the risk has beeri reviewed arid deemed 
worthwhile by those with substaritial ecbribniic iriterests. The level of resources 
needed to develop ari associate for deplbyriierit with a weapbri system is likely to 
be coriiriiensurate with the data cbsts bf the weapbh system acquisition. 
Accordirig to a 1983 Armed Forces Cbmptroller report on weapbri system life- 
cycle cost, 5% bf the acquisitibri cost for a weapbri system is for data (Lahore^ 
198^). However, the "know-how" develbpned iri first efforts will be amortized 
across the succeeding applications. 

Target Erivironmerit and Task 

The target erivironmerit for the.protbtjrpiO^aintairier^ Associate was the 
intermediate-level avionics repair shop fbr USAF F-llls. Figure 2 shows the F- 
111 6883 Converter/Flight Coritrbl Test Station which is used to fault-isolate 
rrialfurictibriirig lirie replaceable units ("black boxes") previously removed from 
aircraft on the flight line. 




F-1 i i ^883 Converted Flight Cbritroi T^st Statibni 



. Automatic test stations such as the 6883 were briginaliy introduced to 
reduce or eliminate the need for manual troubleshbotirig. However, manual 
troubleshooting^is still required to isolate faults which the test station cannot find 
within a unit ander test (UUT) and to isolate faults within the test station itself. 
Although test stations are provided with a self-test capability, most technicians 
prefer to troubleshoot them manually. 

Figure 3 provides a simplified diagram of how an automatic test station 

works by switching stimulus signals through a patch panel and adapter (test 
station interface) to the UUT. Respdnse signals from the UUT flow tock through 
the adapter and are switche J to measuremerit devices which compare the received 
signal to an expected signal. This signal path is termed the "test loop," and there 
isone test loop for each and every test applied to the UUT; The UUT selected as 
the application testbed for the prototype Mairitainer's Associate was the Feel and 
Trim Computer, which is tested by over *00 tests. 

if the test station malfUhctibhs^ this is manifest during a specific test. 

The station would indicate a certain malfunction in the UUT which, when 
repaired, stills checks out "bad." The key to troubleshooting the test station is 
that the probable causes of the test failure are limited tb those components along 
the currentiy^actiye test loop. This is why technicians prefer tb troubleshoot the 
test station manually. The* can Use the current test information to narrow the 
search for a malfunction, whereas the test station's self-test sequentially checks 
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all c6jhi)6nerits. The riiairitainer's associate design was developed to use this same 
test, loop strategy that experienced technicians use to troubieshoot the test 
station. 

STIMULUS to UUT RESPONSE FROM UUT 

MEASUREMENT 

SOURCE -^SWITCHING , ^SWITCHING BEVICE 



1 



TEST STATION INTERFACE 
I 



^UUT / 



Figure 3. A Simplified test Loop for Automatic Test. 

Because the purpose of this R&D was to investigate inteili^e^^ 
mairiteriarice technolo^^ rather than build a system for field use, the diagnostic 
coverage of the knowledge base was limited. The utility of a prototype wq^u^ld be 
dembristrated if the system could fault-isolate from the test station as a whole to 
the next level of repair; that is, to one of the 29 test station replaceable un^^^ 
(TRUs). Achieving this goal would illustrate the fault isolation process through 
brie level of refinement. In order to demonstrate a second level of refineme^^ 
specific TRU was chosen for further fault isolation. (For field use, an associate 
wbuld^ bf course^ continue refinement within all 29 TRUs until the appropriate 
level of repair was reached.) 

The diagnostic coverage of the prototype was also limited to 
troubleshbbting the test_statibh when the UUT was the Feel and Trim line 
replaceable unit (LRU). This LRU represents about 50% of the test station work- 
load. Because the test strategy depended on troubleshooting the test loop,^ the 
apjproach used was cbn text-sensitive to the particular unit under test and its 
associated test program set and set of test loops. 



Goals and ObjectiA^ 

The bverall bbjectiyes of this effort were to conduct exploratory 
research cbncerning the role of an associate and technologies for further 
development^ and to develop and dembnstrate a prototype Maintainer's Associate. 



Exp l or a tory 



The heart of maihteharice is troubleshooting, thus, a thorough 

understanding of diagnostic prbblem-solving was a prerequisite to understanding 
and designing the prototype Maijitaiher's Associate. Further, because the overall 
goal of this effort was the development of an interactive maintenance system, 
both technician and expert system perspectives on troubieshobting were 
investigated. Human trdubleshootin& and its implications for intelligent 
maintenance aidSj was reviewed in a previous report (Keller, 1985). The present 
report focuses on expert system approaches and specif icaily on a knowledge 
acqujsition strategy called "hybrid diagnosis" which uses knowledge about the 
structure of the system under test^ as well as knowledge about fault/symptom 
associations. 

biagnosis is a special kind of problem-solving called "ciassification 

probleni-soMng'' (Clancey, 198*)* in which the problem-soiver selects from a set 
of pre-enumerated solutions. Diagnostic test strategies are either precomputed, 
as in the traditional automatic test equipment approach to diagnostic test; or they 
are developed in real time as a diagnpstic session proceeds, as is typical in the AI 
apprqach. In either case, the set of "right answers" (i.e., the potential faults) 
toward which a successful strategy converges, is known in advance. 

The key to classificatibh problem-s^^^ is hypothesis refinement (also 

termed "establish-refihe"). A fault is isolated to one of a set of probable causes 
at a given level of abstraction ("established''); then, the probable cause is broken 
down into more finely detailed probable causes ("refined"). This process is 
repeated until the fau^ is isolated to a sufficienily small probable cause set 
(Ghandrasekaran, 1983; Tanner & Bylahder* 198*). This strategy is similar to the 
three-level military maihtehahce philosophy of field, intermediate, and dej5ot 
maintsnance. However, even when it is applied within one maintenance level, this 
strategy of "divide and conquer" has diaghdstic power and efficiency. 

'Tfi^ refine step of the establish-refihe strategy calls for selecting the 

one correct intern from_ a set of possible items. For troubleshooting, the 
refinement process itself consists of five steps which, when repeated iteratlvely, 
cony-rge o^n a fault at a given j^^^ abstraction. These five steps are: (a) 
decide whether further diagnostjc refiherheht is warranted; (b) select where to 
make the next obseryatipn based oh maximizing the expected information gain per 
unit cost; (c) identify the expected value at the selected observation point; (d) 
make the qbservationj and (e) deterrhihe the implications of this observation in 
terms of _component blame or innocence. This process may be summarized as a 
cycle of making observations and cdmpUtihg entailments (de Kleer, 1984). There 
are two ways of implementing this five-step refinement process: the 
specification-based or symptom-based approach. 

Specification-bas ed diagnosi s . The specificatioh-based approach, often 

termed"deep reasoning" (also causal, topographic, tbpblogic, or state-based 
reasoning, solves diagnostic problems by reasohihg from a device model 
(Genesereth, 198*). A represehtation of the components that constitute 

a device, together with their input/output behavior and interconnections, enables 
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reasoning directly from a "deep theory'Vconsisting of informati^^^^ 
structure and behavior. Figure ? shows the advantage of the specif icatibh-b^^^^ 
approach. The basic representatioh, the device model, is not specialized fb 
specific task, such as diagnosis, arid therefore can be used for multiple pdrpbses in 
the design and support of weapon systems. A related but simplified form of 
specificatibn-based diagnosis is logic modeling, in which connectivity is modeled 
but riot module iriput/dUtput behavior. 

Iri the specification-based approach, knowledge is represerite^^^ 
propositions that are simple statements known to be true. Exam^^^ 
stateriierits are "the output of the signal generator is connected to the input of the 
oscilloscope" or "the amplifier is bad." Through the use of resolution-based 
theorem provirig (Geriesereth^ 198«), or other techniques (Bavis^ 198*), these 
statements are cbmbiried to develop new propositions, tists of suspected faults 
and tests to be made will have certairi_ forms when represented propositionally. 
The basic idea is to derive these forms from the current set of propositions when 
a list of suspects br a riieasuremerit is needed. 

Using only the device model, the composite behavior of the syste^^^ 
be derived by propagating individual compdrierit behavior through the connectivity 
network (Davis, 1984; de Kleer 1976; Sussman & Steele, 1980). Knowledge about 
this behavior is also constrained by applicable network laws, such as Ofim^s and 
Kirchbff's laws. 

With the specif icatidri-based approach, the device modej of the sy^^^^ 
under test is in the erigirieer's mirid if the diagnostic program is being developed 
directly by a test erigirieer. If the diagnostic program is Al-based, then the device 
mbdel is in a computer. Iri either case, tnis model Js^ used to generate 
expectations about circuit measurements, which are compared with actual 
measurements. Discreparicies betweeri expected and observed values are t^ 
used to rule but certairi cdmpdrierits and cast suspicion on others. As described in 
the five steps of the establish- refine cycle, the new state_of the model is used to 
select the riext measUremerit, based on the maximum information gain. 

Symptom-based diagndsis . the sj^rn ptom-based apprbach, often termed 
"shallow reaidriirig" (also pattern matching, evidential^ assbciatJonistic, or 
empirical reasbning), sdlves diagridstic problems by manipalating a set^ of 
associatibris between symptoms and faults. With this approach, the assbdatibn^ 
between symptoms arid faults represent a compiled form of knowledge which is 
streamlined arid cdriditioried fdr the diagnostic task. The principles and models 
from which this knowledge is derived are not always readily accessible tb the 
problem-sblver br may everi be unknown or forgotten. : Of ten symjytonri-ba^^^^^^^^ 
knowledge is heuristic in riature (i.e.^ fallible) and is based on experience more 
than reasbned causal derivatidri. 

Gerierally^ the associations in the symptom -based approach are f bunded 
bri simple empirical dbservatidns, but they may also be logical consequences 
deduced from the device model of the system under test. As such, these systems 
represerit diagridstic knowledge in a compiled form. Here, the devi model and 
general diagridstic algdrithm are used to compute a special-purpose data structure 
taildred to the diagnostic task. 
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^'fi"*"^ ^ - "lustration of How a Device Model Relates Design Features 
and User Functions. 
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Hybrid diagnostic reasoning ; AI systems have been developed for both 
the specification-based and symptom^based approaches^ Human problerh-sblving 
technicians also use either apprbachji but generally prefer to use shallow reasoning 
when possible and resort to deep reasonirijg only when forced to do so (Rouse, 
1984). a! systems can employ a similar strategy of using both techniques as 
needed but to date they do not, te^^ instead to be one or the other, but not 
hybrid combinations. The two approaches, however, a^re^ inherently interrelated 
For example, there must be a causal explanation for every empirical fact. The 
specification-based approach focuses oh the causal explahatibhi the symptom- 
based, on the known fact. With one exception, described by Fink^ Lusth^ and 
buran (1984), expert systems that capitalize on the potential synergism between 
' e two approaches do not exist. 

Table i provides a sample of literature relevant to computer-based 
diagnosis. Ah ih-depth review of work on speciiicatibh-based_ diagnostic reasoning 
is found in King (1982), and one volume of the iourrial Artificial Intelligence 
(Bobrow & Hayes, 1984) is devoted to qualitative reasoning about physical 
systems, bringing together research previously published in scattered conference 
proceedings. Artificial Intelligence in Maintenance (Air Force Human Resources 
Laboratory, 1984) contains a number of the works cited in the table. 



Prototype Besign_and Beveloprrieirit 

Specific objectivcts for the developmeht _ arid demonstrati^^^ of the 
prototype included: (a) demonstrating a Mairitainer's Associate that serves as a 
skill multiplier for inexperienced techriiciaris arid as a skill integrator that uses 
and captures the corporate memory of skilled techriiciaris^ (b) developing an 
efficient authoring system for developirig the Mairitairier's Associate knowledge 
base, (c) constructing a portable MaintainePs Associate hardware uriit for the end- 
user, and (d) coilecting and analyzing responses from meriibers of a mairitenarice 
organization for use in guiding future efforts. 

Chapter 2 discusses design detail, arid the implemeritatidri of the design 
is discussed in Chapter 3. Chapter 4 describes the knowledge erigirieerirtg effort 
conducted for the prototype. The results of the system denipnstratibri efforts are 
presented Jn Chapter 5. Chapter 6 provides conclusions and implications for the 
entire effort. 
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TafeleJ.. A Sample of Literatdfe Relevant to Com{5Uter-Based Device DiagnSsis 



Diagnostic 
Approach 



Literature Reference 



im 
Name 



Logic Wong and Andre (1976, 1981) 

Modeling Andre arid Wong (1975) 

Lorigeridorfer (1981) 
Cramer et al. (1982) 
DETEX Systems^ Inc. (n.d.) 
Simpson and Baiaban (1982); Simpson and 

Agre(ig83) 
Cantbne (198*); Cantone et ah (1983, 198*) 

Specification- Brb\vn and Sussman (197*) 
based Stallman arid Sussman (1977) 

McDerrriott (1976) 
Brown (1977) 

Brown, Burton, and de Kleer (1982) 
Geriesereth (1982) 

Davis il983); Davis et al. (1982); Hamscher 

and Davis (198*) 
Pipitbne (198*) 

Symptom-based Mcbermbtt arid Brooks (1982) 

Hinehman arid Morgan (198*); Williams and 

Hirichrrian (1983) 
Bonissbne arid 3bhrison (198*) 
Davison (198*) 

Laffey, Perkiris, arid Nguyen (198*) 
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CHAPTER 2: DESIGN OF THE MAINTAINER'S ASSOCIATE 



Three general design issues that were encouritered arid resolved iri the 
design of a niaintainer's associate are discussed in this chapter. First, the issue of 
hybrid diagnosis is considered^ which integrates both specification- and symptom- 
based knowledge in the same knowledge base arid iriterprets both with a single 
inference engine or reasoning strategy. Second^ the target equiprilerit preserited a 
special design challenge because the autorriatic test equipmerit is a recorifigurable 
system; that is, its device model is not static but varies for each of oyer *00 
different test number states. _ The problem of desigriirig for recdrtfigurabre 
systems is therefore examined. Third, desijgn issues related to the user interfaces 
that support system authoring, skill multiplier, and skill integrator concepts are 
discussed. 



Hybrid Diagnosis 

For each level of refinement in the hierarchical decbmpbsitiori approach 
to problem-solving, the compiled test tree spans from a parent node (a deyice 
module at one level of the hierarchy) to a set_of constituent cbmponerit nodes 
(that module at the next level of refihemeht). The test nides between these two 
levels correspond to subsets of constituent comporierits. The diagnostic test tree 
shown in Figure 5 represents the compiled knowledge of a specificatibri-based 
approach to diagnosis which is now in a form cbmpatible with symptom-based 
diagnosis. The tree is essentially deterministic iri character; given various 
outcomes of tests beginning at the root node of this tree^ the prbblem will resolve 
to the correct faulty subcomponent at the next level of refirieriierit. 




Cbmponent 



Coristituerit subcbniporiertts 



Figure 5 . A Specification-Based Teist Tree with ari Overlaid Heuristic Irifererice, 

The Jtrerigth oi the symptbm^based approach tb diagnosis is iri the use of 
heuristics. These are "rules-bf-thurinib" which capture knbwledge deriyed froni 
experience. Although these rules are devjce-dependent^ they often have a great 
deal of diagnostic precision that is not derivable from a structure model. The 
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rules of inference in the symptom-basiBd approach to diagnosis are in the form 
"symptom impHes^ fault." Because there are few or rib limits on what can be 
described as a symptom, the rules can capture quite complex patterns that serve 
a^ signatures to specific faults.^^ these heuristic rules can shortcut several 

levels of diagnostic tests generated by a specif icatidh-based approach. Heuristic 
inferences of this sort can be rej>resehted by an arrow indicating that, given a 
certain symptom pattern, a particular subcomponent is directly suspected to be at 
fault as ^ shown in Figure 3. It is sometimes necessary to backtrack and undo 
diagnostic inferences based on heuristic rules^ because a heuristic is not infallible. 
When this happens, control moves up in the diagnostic tree instead of down, and 
the previous path that did not yield a solution is ruled out from further 
consideration; 

In the design for the prototype, specification- arid symptom-based 

diagnostic approaches were integrated by cbnripilirig the specification-based 
informatkri into a diagnostic decision tree upon which syrriptbrri-based heuristic 
rules were overlaid. This integration capitalized bn the ease of developing a 
diagnostic knowledge base that was characteristic bf the specification-based 
approach^ while at the same time incorporating heuristic knowledge in the form of 
syrnptom/fault associations. Two important bbjectives of this design effort to use 
hybrid diagnosis included documeriting hbw knbwledge engineers build a diagnostic 
tree so that the process can be computer-aided or computer-automated and 
determining the relative proportions of specification- and symptom-based 
knowledge used in diagnosis. The results of these two activities in prototype 
development are discussed in Chapter 



Recon fij^urable Systems 



In designing and developing the knbwledge base^ the following questions 
had to be addressed: Since the test station can be iri any of over WQ states 
(depending on the test number it is executing), wbuld there need to be one set of 
rules in the knowledge base for each of these states? Further, what is the impact 
bf the state of the system under test on the structure of the diagnostic knowledee 
base? 

J^l^s^ questions were addressed by realizirig that whatever changes in 
state the test station goes through, the test loop remains invariant at an 
appropriate level of abstraction. In other words, fbr any current state of the 
station, a signal is routed through the UUT tb a measurement device, as 
previously described Jn Figure 3. Thus^ at the first level of refinement, it is 
possible to view the test station as a number of generic regions along the path of 
the abstract test loop. For each test, the test station is sent a sequence of 
programming jnstmctions w^hic^ set the coriditibns required to perform the given 
test. If it i^s assumed that test station failure is always associated with a specific 
test number, it is then possible to determine the specifics bf the signal path and 
the expected signal values at the various points albrig the test loop. Given this 
perspective, only one generic diagnostic test tree must be developed. 
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this diagnostic test tree is a hierarchy of tests which splits the set of all 
probable causes of failure (represented by the root of the tree) into small subsets 
until a failure can be isolated at the current level of refjnement^ Devel^^^^^ 
tree requires deciding where tbpblogically to measure and the consequences^ Qf^a 
measurement in terms of absolving or blaming cqmponentsi For any specific 
test, these requirements translate intp_ knowing the precise physical location for 
test and the correct signal value to expect. 

there are several alternatives for deriving these expected measurement 
parameters. One approach is to interface th_e J^APert system with a correctly 
functioning piece of hardware. This is the approach taken in signature analysis^ 
A seeondjapprbach is tb query a standard circuit simulator, as is done tn SOPHIE I 
(Btown, Burton, & de Kleer, 1982) or STEAMER (HoHan, Stevens, & Williams, 
1980). As a third approach, a_ devisee model may be used, with expected 
measurement values computed through constraint propagation and dependency- 
directed backtracking (Davis^ 198*; Geneserethj 1984; Sussman & Steele, I98fl)^ A 
fourth alternative is to have subject-matter experts or knowledge eng^^^ 
develop expected values mentally and enter these values into the expert system as 
data, as was done in Pipitone (198<^). 

In the present effort, a fifth alternative was employ^^^ 
expected rheasurerhehts was developed from the test program set for the 
automatic test station and the tabular and schematic information available from 
the technical docurriehtatibh. This file_o:f expected measurements was generated 
by a spiecial computer program called a parser. The parser was designed as an 
editor so that it cbUld be used withj)ther test stations or with other equipment 
with sets of data organized by system state stipulating expected signal values and 
locations. Details bf the parser are discussed in Chapter 3. 



User Interf a ces 



In .order to implement the desired functions of the maintaiher's associate 
concept, DRI designed a series of user interfaces. These interfaces enable the 
technician to be both systerri-builder and end-user, beca,use both are important to 
the successful develbpmeht, use, and maintenance of the database on which the 
system operates. In the fbllbwirig sections, the_ design specifications for three 
optimal user interfaces are presented: an authoring system, skill multiplier 
interfaces, and skill iritegratbr interfaces. 



The Authbrihg System 

The dembhstratibh of tools for developing the maintainer's assbciate is 
nearly as important as the demonstration of tjie prototype itself. Large-scale 
implementatibh bf these devices would be impossibie without the means to 
efficiently develbp^ debug, and maintain their knowledge basesi As it was 
expected that sbme kribwledge base debugging and maintenance would be 
conducted during system operation, it was vital that the authoring tbbls be 



15 

B7 



EKLC 



'"*e.gjated with run-time softvs^afe so that a knowledge engineer or author can 
ti'.^nsfer effortlessly between Using the device and editing its knowledge base, to 
enhance this process, various types of iriforrriation must be visible and accessible 

therefore designed to augment domain-specific 
messages with all other pertinent information regarding the state of the expert 
Sllrmld^ "gl^lox?" '''''''''^ °' information suggested that the editor 

♦ Iwo procedures were designed to implement changes in the state of the 
f^^t^;rI^^ ^^^^^^ single-stepping, in which states shift step-by-step 

'^^^'^^^ '^'^^J^^^^^'Tt system's inference engine. The second Approach 
was interaction-stepping, where the system state is visible as the system bauses 
for user interaction. Because this design allows the system authirti st?^ the 
expert system^ through its algorithm, viewing the resulting states along the way 
the editor is also termed a "runnable editor." ^ ^* 



The Skill Multiplier Interface 

At the minimum, a maintainer's associate must pF5mpt the user for only 

.necessary information and inform the user of the eventual diagnosis. In thil 
r^PiPAl system would operate as a fully proceduralized job pirformance a?d 

recolmi^ S^^^^"^^^ ^^^^^^ P'-°"^°*« active learning nor 

recognize any differences in user competence. In this section, a numblr of 

feamTrjf .I'e'deS^lESi!"'" ^"^^^ned to support on-the-job 

, , T>^e "Ho^^to" interface. The purpose Of this interface is to augment the 
norm|l interaction message; typically it is a multiple-choice question regardins a 
specific sjgnal or test,^ith more detailed information about where to llcate fhe 
signal or how to perform the test. Fo^ example, the interaction frame might ask 
the technician to use a digital voltmeter and report the value. If the te^nidan 
S^il "o? i"?^. do this, the how-to interface would provide details. The 

levjl of de tair could be structured hierarchically so that the user gets just the 

tii^?Z°T l^^^^^^^^^ provided in this interface, as in interactions 

themselves^ combine text and graphics. 

Hi^on^ct^^^^ "where-from" Intprf are. This skUl multiplier interface involves the 

wnS^i^?^^^ '^^^ '^^^'^^^ "^^^^^ °* physical manipulations, the 
where-from function answers the question: "What has happened so far?" tists of 

SttH'^rfrSlK?''^^"*^'^ interactions and their answers, assertions in working memory, 
|nd probable causes would be provided by leVel of refinement. Also, if evidence 
included a special rationale entered by the knot^ledge eSg&er durlg 

throu8h'thi??«tS:"'"'' '''^ explanation «ould be acce3s3 

The "where-to" inteEface. This interface also relates to the diagnostic 

^'■^ y°" "^^^ that question now?" or 
Why^should I conduct this test?" In response to a where-to query, the system 
would explain what evidence may be obtained by conducting this test and how that 
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evidence may help discnmihate arti^^^^^ cUrrerit probable catasesi This information 
would presehted as an English-like reriditibh of the rules of evidence which 
caused the interactions to queue Up. The user would also be able to request to see 
the other tests which queued Up for this level of refinement and their associated 
evidence or any canned messages associated with the evidences 

these skill multiplier interfaces would allow the user to obtain more 

information about the ongoing diagnostic session. This information is accessed, as 
needed, under the user's control and thereby promotes skill development on the 
]9b. This should prove effective because the user is presented with this 
information only when requested and always in context. 

The r efe rences interf ace . The expected values of various measurements 

are provided to the techmcian by the system through message interactions. In 
developing the database of expected values^ indices to the sources of information 
can also be saved, through this interface^ it would be possible f^r the technician 
to access this additional techhical dbcurrientatidn* _Jh answer to a technician's 
question (such as "How did ybU know to check pin XYZ?"), the system would direct 
the user to the appropriate reference for that information. 

^ Future implementations of a maintaiher's associate could extrapolate 

tWs interface to a general cohtext-def>Bricfent index to all technical information 
abqut^the system under test: theory of operation, setup* checkout, calibration and 
alignment procedures, schematics, tables* illustrated parts breakdowns, and 
removal and replacement procedure^^ Having this information stored on-line as a 
relational data base alleviates the two principal shortcomings of current 
documentation: the physical bulk of paper-based dbcurrientatioh and the difficulty 
in finding and cross-ref erericirig heeded information. 

The tuto i^terface^-maihtehance trbubleshbotirig sirhuiation . The four 
skill multiplier interfaces described above were designed to be available to the 
user during a cohsultatidh at any point in the current diagnostic process. In 
contrast^ the tutor interface wdUld be a distinct, special-purpose mode of 
operation that could be selected while the user has some spare time or during a 
time period allocated to formal study._ Iri the tutor mode, the basic consultation 
process would be reversed: Instead of the associate fault-isolating for the user, 
the user would fault-isolate for the associate. Rather than providing input 
requested by the maintainer's associate, the user learns to lead the associate by 
generating the diagnostic steps that it would follow. A strategy would have to be 
developed to avoid potential natural language problems. For example, the tutor 
mijht display a list of probable causes, ihcluding one that does riot belong, arid ask 
the user to identify the distractbr. SLriiilar means of forcirig the user to 
anticipate the associate's processing wbuld be develbped for the other steps in the 
establish-refine cycle, the exact sequence the user must follow, giveri a selected 
fault for maintenance simulation* would be generated by fbllbwirig the path that 
leads from the fault back Up to the rbbt of the diagnostic tree. The tutor would 
build this path bottom-up, and then force the user tb fbllbw it tbp-dbwri. 

1__ In future implem en tatibris, this interface could be linked with the status 

records of the technician's on-the-job training curriculur.u If the objective of the 
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curriculum was to en^ the technician to trouBl^^ known to bccdrj 

the diagnostic tree itself would har^^^ a hierajxhjcal description^ the 

curriculumj i.e,^;^ the te^chnici^an's co^mpet^ence could be modeled as an overlay on 
the diagnostic^ tree with the portions that the technician has mastered marked as 
suchi Then, employing a suitable sequen^dng st tutorial simulatjon 

exercise could be selected in accordance with both the training curriculum and 
the trainee's demonstrated competency^ 

Through the tutor interface, the technician should learn the basic 
establish-reflne approacKto diagnostic problem-solving and the specific structure 
of the solution space. If the technician strays off the tutor's path, immediate 
negative feedback would be provided, justified where possible with the canned 
rationale for evidence rules. The technician would be (aj taught in the context of 
prbblem-solving^ (b) modeled as ah overlay or subset of the associate's rule base, 
tc) instructed in the goal structure of diagnostic problem-solving, (d) have his or 
her working memory load minimized^ and (e) have the exploration of wrong paths 
cut off immediately.. All of the above features have been described by Anderson^ 
Boyle, Farrell, and Reiser (1984) as the functionai prescription for ihteiligent 
tutoring syste s. 



The Skill Integrator Interfaces 

Skill integrator Interfaces would have three functions: (a) to support 
user initiative in diagnostic problem-solving, (b) to capture the corporate memory 
for troubleshooting as this memory develops, and (c) to support routine 
maintenance event reportihgi Three Specific interfaces were designed to 
accomplish these functions for the maintaiher*s associate systerrii 

. The "browse" interface ^ The solution space in the Maintainer's Associate 
can be represented as a structured hierarchy of probable causes^ with some 
indicating specific components and some indicating subsets of components. The 
browse interface would allow a visual representation of this hierarchy, which the 
us^r could peruse. Using a mouse or other pointing device, the user could also 
point to any node in the tree and call up the list of assertions which must be true 
in order for the system to accept that the fault could He in the subtree beneath 
the indicated node. Because more than one path from the root of the diagnostic 
hierarchy to any given node may exist, the list of acceptable facts would be only 
suggestive of what actually may be the case. The user could use Jhe browse 
interface to compare what he or she knows to be true to jvhat the maintadner's 
associate system would accept as true for a given fault at any level of 
refinement. 

The "jumFHahead" interface .^ I^his interface would allow^ user to 
initiate diagnostic refinement at any given node in the solution hierarchy. While 
operating in the browse mode, if the user found a gob what is 

known and a certain probable cause, the consultation could be started at that 
point. In starting at selected nodes^ the Maintainer's Associate would not assert 
the facts it would believei If these facts are subsequently needed, they would be 
automatically substantiated through the normal interaction mechanism. If the 
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user rriade a poor judgment abo at wh^ to begin, the system would eventually 
back up to the user's indjcated starting point and explain that rid further progress 
could be made with this node as the starting place. 

^ The briefing interface . The Briefing interface would have two facets: 
prebriefing and debriefing, Prebriefing would permit access to the maintenance 
informiati^n system's records on the aircraft^ s^sterrij black box, or card under 
tiest. Useful information sueh^as the component's repair history or environmental 
arid mission correlates of the malfunction could be accessed with this interface. 

The_ debrief irig interface would be a gateway to a text file for user 
comments. These comments could be indexed by the nqd^in the i)roL>able cause 
hierarchy at_ which notes vere entered; and users could make comments, about 
any aspect of the interaction with the associate, ranging from apparent knowledge 
base inaccuracies to suggestions for new rules. 

In a sophisticated associate, this interface would not merely accept 
textual input but would actively formatjt in accordance with the comment type. 
If the comment concerns the knowledge base, the system would verify this with 
the user and attempt to formulate the suggestion Jn the semantics arid syntax of 
the rule base. Furthermore, in later developments, the briefing interface would 
riot only accept user comments, but also request theni. For example, wheri the 
user successfully solves a problem using the "jump-ahead" interface, the 
Maintainer's Associate would use the interface to Initiate a dialog^ue to capture 
the heuristic that the technician had successfully applied and which enabled the 
jump-ahead. 

I^ater versions of this iriterf ace could also serve as the technician's 
access pbirit to the ground-based maintenance information system in which data 
about the niaintenarice event are collected and/or reported. The technician could 
Iriput the corrective action, time taken^ and other standard mairiteriarice 
irifbrriiatibn upon the successful completion of fault isolation and repair. 

^ ^As previously rioted, the desi^ features outlined in this chapter provide 
an idealized operationalization of the maintainer's associate concept, those 
features that were selected for Implementation and demoristratioh in the 
prototype system are described in the next chapter. 
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CHAPTER 3: SYSTEM IMPLEMENTATION 



The basic system software arid delivery _hardware for the MairitairieHs 
Associate were deyelbped^y Gerieral Dyriamics^ Electroriics Division^ as part of 
art irtdeperiderit effort. For the.develbprnerit of the Maintairier's Associate 
prdtdtype^Jt was riecessary for DRI to modify this basic software and design a 
parser. This chapter describes the additional software development and 
riiddif icatidrts^ as well as the original software and hardware^ 



System Hardware 



Software develdpnierit arid riile base authdririg >vere accbriiplished on a 
Xerox 1108 persdrtal wdrkstatidrt (Iriterlisp-Pl^ cdrifigured with 1.5 megabytes of 
main rtienldry arid a *3-megabyte hard disk. ThiB display. was a large-format CRT 
(17" diagdrial) with a high-resdlUtidri bitriiap (102* x &D8 pixels). The delivery 
hardware provided by Gerieral Dyriajnics^ Electronics Divisibri^ was a 4>6rtable, 
battery-dperated^ briefcase-sized unit terriied the "box." As shbwn iri Figure 6, 
the bbx hbuses the battery pack, main prbcessbf (Iritel 808^), 1 megabyte of 
rartddm access meriibry^ arid a reriibvable display/iriput uriiti The battery pack is 




Figure 6 . The Maintainer's Associate Portable Unit on F-Hl Test Statibri 
Wbrktable. 
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capable of supporting 10 cdhtiriUbUs hbUrs of operatibh. The display/input unit is 
approjcimately 5" by 8"; has ah electroluminescent screen with a resolution of 
256 X 512 pixels and a 16-elemeht keypad next to the screen, consisting of the 
digits 0 through 9; single keys cdrrespondihg to the user interfaces WHERE^TO, 
WHERE-FROM, HOW, MARK & RETURNIj_a key^ to move forward, labeled NEXT; 
and a key to move backward, labeled BACK._^bftwarejs downloaded from the 
development system into the box via IBM PC arid RS232 connections; DRI's 
completed prototype Maintaiher's Associate occupies a total of ii8K bytes of the 
box's memory, including 23k bytes for the run time software, 13K bytes for the 
knowledge base arid associated graphics^ arid 59K bytes for a file of expected 
measurements. 



Supportirig Software Eriviroririierit 



itecture 



The expert systerti shell used for this project is Ruie-Kit (General 
Pynamics, Electrbriics Divisibri, Rule-Kifs architecture, shown in Figure 7^ 

uses classificatiori prbblem-sblvirig^ the establish-resfirie approach, and a 
knowledge base cdrisistirig of a dia^bstic hierarchy. Each node in the hierarchy 
contains a list of successor ribdes^ iritb which the parent is refined, and a set of 
rules of refiriemerit called "eviderice rules." 

The basic Rule^Kit algbrithrti has as its bbjective, at each level of 
refinement, picking a "wiririer" f rbrii the isuccessbr nodes using the evidence rules 
contained within the parerit ribde. This list of successor nodes is termed "the 
refinement list." the eyiderice rules Scribe weights to members of the 
refinement list, based on the existerice of certairi facts in working memory (the 
collection of facts developed duririg the cburse bf a diagnostic sessibh). 

The first step iri this prbcess determiries the existence within wbrkihg 
memory of a fact which will cause one of the eviderice rules to fire, thus assigning 
a specific weight to brie or rtibre members bf the refineriierit list. After all of the 
eviderice rules have beeri scariried arid matched against memory, the refinement 
list is examiried tb see whether or riot one bf its members is now a ''winner" 
(defined as havirig ari accumulated weight bf IBQ or more points). If there Is a 
winner, theri the refineriierit process begins agairi, using the winn^er as the 
be refiried.^ If there is rib wiririer^ the eviderice ruies are scanned again to index 
corresppridirig interaction frames which are used to request In^formation frorn the 
user. After all the iriteractibn frames have been c^ijertedj they are prioritized 
accbrdirig to pbteritial infbrmatibn gairi. TH^ is com^pated as t^ total points for 
all iriterac^tibri frame butcbriies ascribed by applicable evidence rules Lo members 
of the refiriemerit list^ divided by the cost of running the test and the number of 
butcbmes. 

The riext step iri the prbcess is to run the first i^ter^ frame on the 
priority queue. At the conclusion bf the interaction, a fact is asserted in working 
memory cbrrespbridirig tb the new information developed, this fact Is now 
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matched against the evidence rules arid the appropriate rales fire, thus ascribing 
new points to the members of t^e refinement Hsti This process is repeated until 
one of the members of the refinement list is a winner or there are no more 
interaction irames that can^ be run (given a winner, the refinement process 
continues). If there is hb winner^ however, that member of the refinement list 
with the most accumulated points is selected; 



When hd further refihement is possible^ it is necessary to determine 
whether or hot the refined compbheht is indeed responsibje for the failure. If not, 
Rule-Kit backs up the diaghbstic decision tree to a point of uncertainty and 
selects a different path from the one that led to the jnaccurate diagnosis. By 
storing the refihemerit data in an audit stack, movement backward through the 
tree is cpritrblled sijnply by popping data off of the audit stack. The degree of 
backtracjcing required is determined by popping the stack until a decision point is 
discovered which had no clear winner (iiCi, no element in the refinement list with 
at least 100 points). The successor node that had been chosen is then eliminated 
f rom the refihemeht list, and the diagnostic process resumes at this level. 

The Rule^Kit_sbftware emplb^^ in the MalntainePs Associate project 
consisted of (a) a Rule-Kit development system, (b) a I^ale-Kit delivery system, 
(c) a Validatbr-Verifier,^a^^^^ (d) a graphics workstation^ Each of these elements is 
briefly described in the following sections. 



_ The development system software provides for editing and running a 
Rule-Kit applicatibh. It is written in Lisp and has been ported to a number of 
diffefent machines, ihcludihg a Symbolics 3660, a Xerox 1108, and an IBM PC-XT. 
Although the versions of Lisp differed for each host system, the application 
knowledge base was cbrnpletely portable since its syntax is invariant (simply an 
ASCII text file). The development system consists of the Rule-Kit inference 
engine and a set of commands used to run consultations and to build or edit the 
knowledge base. 



A streamlined versibri of the Rule-Kit software enabled running 
consultations ph the portable hardware unit; This software was written in the "C" 
programming language and bccupies^ 23,552 bytes. Knowledge bases developed on 
the developmeht system were transferable, without change to the rule syntax, to 
the run time (deliverjr) ehvironmeht for execution by the Rule-Kit run time 
system. The run time systerri compressed the knowledge base file in order to 
minimize memory space usage in the portable hardware unit; 

Validator-Verifier 

the validator-verifier is ah automated version of the Rule- Kit inference 
engine. Its purpose is tb take an existing application (knowledge base) and 
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exhaustively construct all paths from each initial symptom, whhi^^ a given range 
of ^ocUs,_tb its terrriihatirig probable catisei For each path, an audit trail is 
rhairitaihed that cbntairis pertinent information used in constructing the line of 
reasbhihjg from iriitiaj symptom to terminating prbbabie cause: the interaction 
frames examined^ the answers selected^ the level of consultation, the assertions 
made^ arid evidence points given to probable causes. In addition, the vaiidator- 
yerifier labels all assertions according to supporting evidence linkage, whether 
currently linked (evidence exists at current level), later linked (evidence exists at 
lower level), or hot linked (ho evidence at any level). All audit trails are saved for 
ihterpretatibh by the user through the use of a number of analysis functions. 



Graphics Works tation 

This facility supports the constructibh bf graphic images displayed In 
cbhjuhctibh with ihteractibh frames^ Using a graphics table and user-friendly 
menu of bptibhs^ graphics with assbciated text are rapidly developed, scaled, 
edited, and saved fbr use. Graphics are pNdstprocessed by a data compression 
routine to minimize memory usage. 



System Development 



The Denver Research Instittate developed three related software 
elements for the prototype Maintainer^s A^s^ociate: (a) the eENPA 6 parser; (b) 
modifications to the existing Rule-Kit, specifically the glass box editor; and 
user interface features. In order to explain the use of the prototype and the 
software^ a scenarib which illustrates typical troubleshooting procedures is 
presented. 



A Trdubleshbbtirig Scenario 

The setting for this scenarib is an F-lll intermediate-level maintenance 
shop* A^ faulty UUT (in this instance a Feel & Trim ebmputer) is delivered by 
flight line personnel for diaghbsis_ and rejjairi The technicifiSi begins 
trpubieshbbtihg by connecting the UUT via cables to the 6883 te^^ 
initiates the approjjriate automatic testing sequence^ The test station, under the 
cbhtrbl of a CENPA^ computer, performs a series of ^ests on the UOt, e^ 
designed tb test a specific cbmpbhent bf the UUT. Assume fot^ this scenario that 
the testing sequence halts at test 301982. This test failure seems to indicate that 
the rrialf unctibh has been located. At this point, the technician disconnects the 
faulty UUT and re-runs test 301982, ihis time using the shop standard UUT known 
to be in perfect wbrking condition^ The test fails again, thus isolating the fault to 
the test station itself rather than the UUTi 

In a typical maintejnarice shop, the technician would nq^^ use the 
technical brders and common manual test equipment^ to pinpo^^ 
the assistance bf the Maintainer's Associate, however, the technician is aided in 



25 

37 



EKLC 



this further trpubleshbotihg^ process. The Maihtaiher^s Associate asks the 
technician to make a series of tests and report the findings^ and ases the answers 
to help isolate the malfUhctij>n._ To isolate the faulty the MaintainePs Associate 
uses data generated by the CENPAC parser. 



The CENPAC Parser 

The CENPAC parser was developed in order to provide important state- 
specific data to the run time Rule-Kit software. At the beginning of each 
consultation session^ the Maihtainer's AssociatC'-asks the technician to enter the 
test number at which the test station failed. Based oh the test number, the parser 
places in working memory the set of instantiations (expected measurements) for 
each generic region of the test station. For the 301982 scenario^ a n um^ lists 
are placed in working memory, each of which includes: the generic region which 
serves as the key for the match (e.g., STIMSOURCE-OOTPtlT), the signal value 
expected to leave the region (e.g., .08 Hz i^Q VOLTS MOD-SIN-WAVE), and the 
location for measuring the signal (e.g., A4AAJ* PINS A B). A^A^ is the reference 
designation used by the 6883 test station documents to denote the signal 
generator. 

These lists of irifdrmatidh are used in the following way. In the Rule-Kit 
interaction frames, all references to the test station are made in terms of generic 
regions. The interactidri frames for these regions contain variables in the 
message template which are bound by matches to the working memory just before 
the interaction frame is run. For example, an interaction frame might ask: 
••Check the output of the stimulus source at SIGNAt-fceeATlON for this signal: 
SIGNAL-VALUE. Is the signal correct?" When this interaction frame is invoked, 
working memdry is scanhed for a_ match on the region associated with this 
interaction frame, STIMSOURCE-OUTPUT, When the match is found, SIGNAL- 
LOCATION and SIGNAL^ALUE are replaced in the Jmteraction frame me^^ 
with the specific signal Ideation (i.e., A»A»J» PINS A B) and the specific expected 
signal value (i.e., .08 Hz ».0 VOLTS MOB-SIN-WAVE) , so that the message now 
reads: 

••Check the dutput of the stimulus source at A4A434PINS A 
B for this signal: ,.08 Hz *.0 VOLTS MOD-SIN- WAVE. Is 
the signal cdrrect?^^ 

The generic region instantiations that are_generated by the parser could 
not be derived directly by decoding the test program set for a given test number 
as technicians dd. Instead, the information had to be derived from the 
accumulated state df the test station at the start of each test. To illustrate how 
the test program set for juccessLve test numbers yields all the information 
needed, cdhsider the example scenario once again. For test number 301980, which 
precedes 301982, relay 10/1 is set to route the stimulus signal to the tJUT; and the 
generic region instaritiatidhs for thjit test include pins and test points associated 
with relay 10/1. In test 301982^ relay 05/2 is set to route the stimulus signal to 
the UUT. As tests are run sequentially until all of them complete without error 
or until the testing sequence halts at a failed test, the generic region 
instantiatidhs fdr 301982 include pins and test points for both relay 05/2 and relay 
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lO/l. Relay 10/1 was hot reset after 391950^ and is still available to route signals 
during 301982. The parser handles this problefm of state accumulation by breaking 
the code trahslatioh process into two steps: (a) decode the test program set In 
order to identify the major devices _used in the test and the value of the signal 
routed to and coming put of the UUT; and (b) use the components identified in the 
first step to pinpoint the signal path used in the test, thus enabling test locations 
to be identified along the signal path. 

Figure 8 provides a detailed description of the^ parser process, the test 
program set is decoded using encoded definitions from the technical orders as 
seen on the left-hand side of the figure. The decoded test program set is used to 
•mulate the test static Finally, encoded technical orders tables 

and a list of abstract regions are matched against the test station configuration to 
yield test locatibris and expected signal values; This process is described more 
completely in the f dllbwihg sections. 

Decdde the test prograrh set^ The first step in parsing, the decoding 
process, is relatively straightforward. For the 6883 test station, the test program 
sets are represented as hexidecimal codes an^d subcodes, which are easily 
distinguished. Once a code is recognized, it is simply a matter of looking up the 
code and its subsequent subcodes in the appropriate technical orders table to 
obtain the trahslatiph. For example^ a portion of the test program set from test 
301982 looks like :his: . . . 325100 131025*325100 *j*058* . . . For each set of 
six characters^ a trailing indicates that the set begins with a new cqdej thus, 
in the example, 13 is a hew code and 1025325100 are the assocla^^ subcodes. 
Subcodes always follow the code to which they pertain, and the discovery of a 
code in the seqUeritial code/subcode string indicates that a different table must be 
used. Figure 9 illustrates the translation process. Using the proper technical 
orders table,^ it can be seen that the code/subcode string shown previously, 
131025325100, provides the fbllowing instruction regarding 6883 configuration: 
"Set stimulus relay 05/2^ transfer signal directly;" 

This decoder is not device-spedfic; that js, it^^ specific 6883 

knowledge. Both the test program sets and the technical orders tables associated 
with each code are viewed as datai This means that with the addition of the 
proper code trajlslatibr^ tables, the decoder can be used for other test stations or 
other state-depehderit equipments 

Identify the sijainal path. The second pa^rser component, the state 
accumulator, contains sbrhe very^ specific, domain-dependent kno The 
accumulator uses global variables to keep track of the system state; these 
variables cbritairi ihforrriation about active stimulus sources, set or reset stimulus 
relays, current response relays, and currently active measurement devices. For 
each (sequential test number^ the decoded test program set Js used to change the 
appropriate global variables. In this way, the accujTiulator identifies the test 
station cornpbhents in use during the test. The components are then majjped 
across those generic regions whijdi will be referenced by interaction frames in the 
prototype Main^airier's Associate. The final step is to identify the pins and test 
points associated with each region. A technician would do this by looking up the 
various components in the appropriate technical orders tables and/or schematics; 
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Address/ 

Subaddress Information 
Unit Code Code Function 



Relay group: tens digit 
Relay group: units digit 
Stimulus relay set 
Stimulus relay reset 
Transfer directly 
Transfer on program control 
Transfer on time limit 
Transfer on go 
Transfer on low 
Transfer on high 
Transfer on on-off response 
Transfer bri rib-gb 

'tff i 

'l 3* To^ ^5"32"5? Set stimulus relay 05/2^ 

Transfer signal directly. 

Figure 9 . Illustration of the Translation Process for Automatic Test 
Programming. 

the accumulator works the same way. Using indices derived from the identified 
components, the appropriate tables yield signal location informatibn for each of 
the generic regions. Nearly 30 technical orders tables and schematics have been 
encoded for use during this process (see Appendix A) . 



It was orginally thought that the accurnulator cbUld obtain all necessary 
information (input and output signal values^ signal sburce, stimulus and response 
relays, and the measurement device for the test) frbm the decoded test program 
set. However, although it is true that most of the tests use all of these 
components, there are some tests which use only the signal source and stimulus 
relay(s). Still other tests dp use all the cbmporierits^ but they are identified in the 
test program set for a test which has already been run. Further, the expected 
signal value for the response side of the test loop is hot always present in the test 
program set. To handle these variations in programming, specific rules were 
developed. 



WiltiplB stimulus souzce s . There are two types of sijnal sources for 
the 6S83S hardwired signals whose voltage never varies; and signals prbduced by a 
test station device, such as the Ratio Trahsfbrmef or Signal Geheratbri whbse 
voltage is determined via the decoded test program set. Hardwired signals are 
simply power sources which are connected tb a stimulus relay. The relay acts as a 
gate to halt the flow of current br allow it tb pass thrbugh tb the UUT. When the 
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relay is set, the hardwired signal source is active; when resets the source is not 
used. In this way a hardwired signal source can be set in a given test^ bat used in 
many subsequent tests. The state accumulator uses each active hardwired signal 
for each test until a com man di^ ehcountered (from the decoded test program set) 
to reset the associated relay, thus rendering the source inactive. 



Jjl the test station there are several devices that are activated by the 

appropriate setup commands arid that may be used to generate a signal. In 
addition, each source is cdhhected (just like the hardwired signal) to a specific 
relay. The relay must be in the proper set or reset position to allow the current 
to pass freely. A given test uses only one signal input from a given signal- 
producing device; however, a single test program number may generate numerous 
source/relay combinatidhs. For example, test number 301982 instructs the Signal 
Generator to generate three separate stimulus signals, while the test Itself makes 
use of only the first signal. The accumulator keeps track of each signal 
separatbly, using the first signal to set the test station state for the current test, 
the second for the next test^ and the third for the next test after that. In 
addition, the signals so generated remain active as long as the associated relays 
remain in the proper set/reset p^^ This means that the three signals 

generated in test 301982 may be reused several times, as^ with the hardwired 
sijgnals. The difference is that only one of the three signals may be used for each 
test, and the accumulator must Use them in the proper sequence: first, second, 
third, first, and so on. 

, cases, both signal types are present in the same test. Any 

experienced technician would trbUbleshobt the paths designated by each of the 
source/relay combinations. The Maihtainer's Associate must provide signal 
location and value ihfdrmatibri for all paths in order to properly duplicate the 
human troubleshoptihg process. This Is accomplished by providing several 
instantiations for the generic regions^ all of which are associated with the same 
test number. Test 301982 cbritaihs four such signal sources: the FCS Power 
Supply, the Signal Gerieratbr, a hardwired signal routed through relay lO/U and 
another hardwired signal routed through relay D5/0. The accumulator recognizes 
those cases when more than brie signal sburce Js present and generates a set of 
instantiations to represent each sburce. Rule-Kit, in turn, provides the 
mechanism to query first one set^ then the next, until the malfunction Is found. 



Variatio zis in res poiise sigzml desi^Tiatibh . The majority of tests 

set a response relay which routes the output of the UUT to some measurement 
device. For these tests, the respbnse relay and measurement device(s) are 
identified by the decoded test program set. Identification is straightforward and 
the accumulator needs rib specialized rules to set the state of the test station; the 
signal value is calculated Usirig upper and lower limit values found in the test 
program set. There are a riUriiber bf ways, however, that this scenario may vary. 
In some cases, the respbrise signal is not routed through the test station. 
Trqubleshooting such a test requires sijrial location and value Information for the 
stimulus side only (from signal sburce to UUTl,^^nd the accumulator need not 
attempt to identify the respbrise side (from UUT to measurement device). In 
anothe]^ case, no response relay br measurement device is designated in the test 
program set because the devices frbrri the previous test are reused . Still other 



30 

43 

ERIC 



cases require complex arithmetic maneuvers bh the part of the accurtiUlatbr fer 
the technician) in order to determihe the signal value cbmihg bUt bf the UUT. 
These complex cases were eliminated frbfti the present prbtbtype deveibpmerit 
effort, 

Data >^s, knowledge: the final parser output . The generic regibris Used tb 
capture the state bf the test station fbr any one test are hot always used fbr 
every test. In other wbrds^ although *0 pbteritial regibris have been ideritified^ for 
any given test bhly a subset bf these may be Used, The trbUbleshbbtirig strategy 
inherent in the Mairitairier's Associate^ hbweyer^ successively refines a problerii 
space comprised bf all regibris. This makes it riecessary for the parser tb 
provide iristahtiatibri data f or all regibris^ ribt jUst thbse which are Used iril the test. 
Fbr regibris which are Used in the test^ the signal Ibcatibn ari^d irifbrmatibri 
are butput in the fbrm described earlier (REGION-NAME SIGNAL-LOCATION 
SIGNAL-VALUE). Fbr UriUsed regibris^ the system is specifically alerted tb the 
assumption that a regibri that is riot iri Use cariribt be at fault. Therefbre^ the 
parser output is ribt data but kribwledge. The parser prbvides the actual fact that 
wbUld have J^eeri asserted by Rule-Kit had an iriteractlbnLlrame been rUri absblvirig 
the regibri. This fact is bf the fbrrh (REGION-NAME TEST OK)._ When the Unused 
regibri appears iri the refirienierit list dUririg a cbrisUltatibri^ this fact already exists 
iri wbrkirig membry via gerieric regibri iristaritiatibri at the start bf the 
cbrisultatibri. the fact causes ari eviderice rule tb fire^ absblvirig the regibri bf 
blame withbUt havirig asked the User tb niake a test. 



The^^ass Box Editor 

The glass box editbr bperates cbricUrreritly with RUle^Kit cbrisultatibri 
sessions. Iri develbpirig the editor^ DRI prbvided the ability tb add, change, or 
delete data iri the kribwledge base by suspending cbrisultatibri and eriteririg an 
editing erivirorimerit. When editirig is complete^ consultation is resumed with 
kribwledge bf riew data arid the changes. 



When the editbr is irivbked, the screeri is cbniprised bf three wihdbws: 
the cbrisUlt window, the refiriemerit wiridbw^ arid the test qUeue wiridbw. Figure 
10 shbws all three editing wiridbws^ as well as twb other wiridbws the User may 
access for further irifbrriiatibri: a text wiridbw arid an eviderice wiridbw. The 
cbrisUlt wiridbw shows the iriteractibri franie riiessage which is seeri by the User 
dUririg rUri time. The refirienierit list arid test queue wiridbws, which are available 
with the cbrisult wiridbw, are alsb displayed. When activated, these twb wiridbws 
prbvide the ability tb edit the assbciated prbbable caUse rUles arid iriteractibri 
frame tjertl plates. The refirienierit wiridbw alsb prbvides the rtiechariism by which 
the eviderice rUl^s assbciated with a selected probable caUse cari be displayed br 
edited. Firially, the text wiridbw is activated wheriever riecessary tb display br 
edit text, sUch as the text niessage iri ari iriteractibri frame. 

At sbme pbirit iri a cbrisultatibri, the User rtiay sUsperid the reasbriirig 
process by eriteririg the wbrd "sUspericf iri resporise to the prbrtipt frbm ari 
iriteractibri franie. Once sUsperided, the riibUse processbr keeps track bf ariy 
mbvemerit br actibri. If the riibUse rtibves br a bUttbri is clicked^ the nibUse 
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prbcessbr initiates the appropriate action to be taken by the editbri Editing 
continues until cbrisultatibri is resumed via a mouse ebmrriandi This cadses the 
refinement list and test queue to be regeheratisid and thef interaction frame at the 
top of the queue to be run. 



User Interfaces 

the normal Rule-Kit cbnsultatibh rribde^ in which the user responds to 
questibhs presented, iri ihteractibri frames^ was augmented with four user 
interfaces as part of the design process (see Chapter 2). These interfaces were 
partially implemented. In the _p^ Assoetatei _ The four 

interfaces are titled HOW^ WHERE-FROM, WHERE -TO, and MARK & RETURN. 



HOW . The HOW interface accesses engineering data and other support 
ihfbrmatibh^ such as removal and replacement instructions and alignment or 
calibration prbcedures. Selectiba of the HOW key on the keypad provides the 
technician with a brief. explanation bf the required procedure. For example, 
Figure 1 1 shows tKe HOW display for setting up the digital voltmeters The 
principal fbcus bf the present effort was the development and use of diagnostic 
khbwledge (trbubleshbbtihg procedures) erriploying AI methods, not on the 
develbpmeht and display of procedural information. Separate projects in the IMIS 
prbgram (the Gbmputer-Based Maintenance Aids System and the Portable 
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F4gure 11 . Sample HOW Frame from Maintalner's Associate Display. 
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Gompater-Based Maintenance Aids System) Have investigated presenting and 
^splaying this type of informatidn. Therefbrej.althbugh access to this type of 
infqr^rnatiqn was incorporated in the prototype Maihtaiher's Associate, only two 
ajustrative HOW frames were created: one for setting up the oscilloscope and 
one for setting up the digital voltmeter. 

i-FROM. In the prdtdtype^ jthis interface was implemented as ah 



Xaii of the consultatidn in process. By pressing the key labeled WHERE- 
FROM on the keypad, the technician is shdwri the path of the consultation up to 
that point. When Rule-Kit establishes and refines a lavel because the correct 
assertions already exist in working meffidryj the techhieiari does not see an 
interaction frame. The audit trail reCdrdl the interpretive process regardless of 
the presentation of interaction frames. Thus^ the parsesr's assertions may cause 
the interjareter to find a winner withdut having to ask anything of the technician 
and the audit trail may include entire establish-refirie cycles not apparent in the 
consultation's series of interaCtidri frames. 

Figure 12 shows twd sample screen displays created by the WHERE- 

FROM option. The screen is drganized by levels of refinement. The first level is 
a list of ali assertions made by the parser at the beginhihg of the consultation; 
that is, those facts that were asserted^ based solely bh the test number at which 
the fault was manifest. Each subsequent level of refiherriwnt presents ihfqrrhation 
cqrrespqnding to the interpretive prdcess bi the riule-Kit inference engine. For 
example, in Figure 12 these include the refinement list (a list of prqbable causes); 
the interaction frame td be rUri, tdgethtr with a list qf the pqints ascribed td 
prqbable causes fdr each dutcdme; the duicome of the interaction frame; and the 
name of the winning frame. 

, ^HERE-TO. Selectidn df the WHERE-fo feature results in a display 

which contains informatidn dn the expert system's pending processes, as shown in 
Figure 13. This interface is intended fdr situatidhs in which ah ihteractibh frame 
asks the technician td make a measurement and report the j-esults.^ If the 
technician has questions (e.g., "Hdw are you going tb use that information?" or 
"Why are yqu asking me that questiqh now?"), this display explains the name of 
the current interaction and its Cost; the name of the currently instantiated 
parent; and a list of the possible butcbmes of this test^ along with the number of 
pqints to be ascribed tq each probable cause. With this information, the system 
states:"! have determined the fault to lie within this probable cause and the 
fqllowing test is requested tb help me assign pbihts tb each of these probable 
causes now under suspicibn." 

^JARK & RET URN . This interface is a first-approximation of a 

capabili^ of the MaJntainer's Assbciate design which enables the user to j'brqwse- 
and-jump-ahead." When activated by pressing the MARK & RETURN key on the 
keypad,^ a small flashing bqx appears in the corher of the display to indicate that 
the interface is engaged. MARK & RETURN functions like a bbokmark. It allows 
the technician to move ahead in the consultatioh and ahswer questiqns without 
actually making the measurements. JSuring this browse bi the cohsultation 
process, WHERE-FROM and WHERE-TO are also active. Thus, in each 
successive interaction frame, the small flashing box lets the technician know that 
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CHAPTER i: KNOWLEDGE ENGIIMEERING 



Tntrnriiirtinn 



Knowledge engineering has been described as more of an art than a 
science. This is doe to the comp^ uhderstanding, and 

representing the knowledge of experts in the development of a computer-based 
system, it is no sarprise that the^task of knowledge acquisition (i.e., extracting 
knowledge from an e^^ bottleneck in the development of expert 

systems^ A ttem^pts have been mad^ structure the knowledge engineering task, 
to automate the process with expert system development tools and CAD/CAW 
processes, even to eliniinate it by deve^^ that would 

allow subject-matter experts (SMEs) to develop rule bases for expert systems 
directly, 

In this chapter, the knowledge engineering process used by DRI in the 

development of the Maintainer's Associate is described, Fpllowing a brief 
description of task purpose and overall approach, this chapter presents the 
spe^cific j5rocesses involved in the two-levd development of the Maihtainer's 
Assodate rule base and discusses the results of the effort and its relation to 
relevant knowledge engineering issues. 

Background 

^f^K^ tend to rely on symptom-based approaches to 

solve troubleshooting proble^^ approach, in which the 

technician looks for a match between the symptoms of a past fault and the 
curren^t symptqmj is easily into expert systems. Although more 

time-consuming, specification-based methods are usually more accurate. Relying 
on an analysis of the func^^^ structure of the device under corisideratidhi 

specification-based methods include exhaustive searching for faulty compdhents 
(practical only for relatively small sets of possible faults) arid refererice to a 
normal functioning model. Using the latter method, the techriician coristructs a 
n^^n*AL"l^^^i tj^^^ arid develops a set of 

hypotheses like "what would happen if," then discrepancies between the device 
undej- Jest and a normal functiom^ device lead to Ideritificatidri of faulty 

components. Other methods, including half- split, bracket, arid Uricertairity 

reducti^on t^echniques, are en^^^ to basic methods and help reduce the 

number of suspects in the ambiguity group. 

As discussed in Chapter 1, electronic troubleshooters use both heuristic 

(symptom-based) and analytic (specificatiori-based) approaches to isolate faults in 
equipment. Expert systems, in general, have used one or the other approaches as 
a base for the^ development of the rule set. One goal of the kriowledge 
engneering component of this project, therefore, was to capitalize on the 
interrelatedness of^the two approache^^ arid use both in the developmerit of the 
rde base. The method was to compile specif ication--based kriowledge irito a 
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diagnpstic test tree arid overlay syrriptorri/fault information as additional branches 
or tests* 



Assamptions 

The troubleshooting envirdhrTierit for this prbtbtype is a complex 

environment. Electronic mainteriance equipment is inherently complex, and new 
developments in equipment design have increased the complexity. In order to test 
the concept of applying expert systems to this environment, several assumptions 
were necessary. 

Si ngle fault ass ymptidn. In the real world of automatic test equipment 
(ATE) modeled in the protdtype, faults in the electronic assemblies and 
subassemblies can trigger secondary faults which must be located arid repaired 
before full system capability is restored. Develbprrient of a rule base to account 
for all possible combinations of laults was cdrisidered a task too expansiv€5 for the 
present effort, therefdre^ an underlying. assumption of this task was that only a 
single fault is present in the equipment. Identification of that sirigie fault was the 
objective of the diagnostic tree. 

Noninte imUtency assumptidri. A second assumptibri was that the fault 
manifest in the system is hpriintermittent; that is^ it Js a hard fail which Is present 
at the same point on each rerUn df the test sequence. Ari intermittent fault, 
which may or may riot be present on repNStitive l^ts, leads to inconsistencies In 
the diagriostic process. Althdugh intermittent faults are commbri arid present a 
serious malnteriarice prbblem^ the expert system developed for the Mairitainer's 
Associate was not intended td address this prdblem. 



Xes t number as sdciated fault asslimptibni A sequerice of tests is 

automatically run by the ATE and interrupted upon discovery of a failure in the 
process of automatic fault isblatibri. Each test in the sequence of tests is 
identified by a test number. It is possible that a failure of a component can occur 
at any time in any part of the ATE^ even when that component is riot involved in 
the particular test which is being run. An occurrence of a iault urider these 
conditions would be extremely difficult tb identify^ since the expert system 
developed for the Maintainer's Associate is keyed on the dcscbdirig of the 
programming associated with the test number which fails. Therefore, these 
haphazard faults were ridt Cdnsidered in the system; only comporierits associated 
with the areas of the ATE which are currently being used for a specific test were 
considered in the suspect set. 



Approach 



In order to accelerate the kridwledge erigirieeririg prbcess, the proposed 
problem domain was divided iritb twb Jeparate levels^ and rule base develbpmerit 
proceeded at each level in parallel. The first level consisted of the rules which 
isolated the malfunction to a particular TRU in the 6883 test station; the second 
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level focased on the id^^^ of a specific faulty cbmporierit within the 

Flight eontrol Simulator (FCS) adapter TRU. Because levels are hierarchically 
related (Level H is determined by Level I rules), a generic approach was adopted 
for Level I which allowed independent Level II development. Thus, two sorriewhat 
different knowledge engineering strategies were employed. 

Both approaches r^eHed on fairlj sources of knowledge, 

primarily technical documentation and SMEs. the two SMEs for this effort were 
7-skill-level avionics instructors from Air Training Command who were 
responsible f^Dr 6883 aad related automatic test eguipment (ATE) at Lowry Air 
Force Base, Coloradoi Two supervisors from the intermediate-level maintenahce 
shop at Cannon Air Force Base, Nlew Mexico, also contributed technical 
expertise. AH of the SMEs were responsible for providing information oh 6883 
operation, troubleshooting strategies, training objectives, and field maintenance 
activities. 



Level I Development 

tevel I consisted oi^^^ rules and interaction f rames necessary to isolate 
malf unctions to the TRU level. This task required: (a) development of a test loop 
wWch would be applied for all troubleshooting scenarios; (b) idehtif icatiori of a 
troubieshooting strategy which reflected the inter dependencies and 
interconnections of the components of the test station, the training and 
performance objectives qt\d^^^ dri-the^jdb training, as well as skilled 

tec^nidan rdes-of- thumb w and (cj development of a rule set 

which captured all of the necessary information into the Rule^Kit architecture. 

^ test loop shown in Ch^ 1, Figure 3, Is a simplified model of the 
ATE whidi presents signal source, signal switching, routing to the unit under test^ 
and signal measurement. For the development of the rule base for Level I, a rriore 
complete and complex representation of the ATE was required because 
troubleshooting to the TRU level required representation of the possible set of 
TRUs in the schematic, as well as the interconnection of the TRUs via cabling and 
wires. 

^Mjb enhanced t^st^ ^^^^^ from technical documen- 

tation, SMEs, and prior knowledge the ATE. Several schematics of 
the ATE were available in the technical orders; but none presented the 
test station In a way which would easily lend itself to development of 
a troubleshooting St rate^^ or to development of a set of rules. The 
existing schematics were, there^^ synthesized into a single sche- 

matic jdth sufficient cot to adequately represent the ATE as 

perceived by ot SME, whil^^ confining the level of complexity to that 
^ch would be fsmiiiar to the target audience. This synthesized test 
loop schematic was showi to both Instructor and field SMEs in order to 
define the relations of the test loop areas j^aiich represented specific 
^tJs and their interconnections. The final test loop schematic is 
shown in Figure 14. 

Establishment of the test loop schei^ati^c jed to development of a 
troubleshooting strategy which reflected the troubleshooting approaches of all 
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four S ME s. Reaching a consensus regarding troubleshooting starting points and 
test/measuremeht sequences was critical to the process. When discrepancies 
arose among _SMEs^ justification jvas sought for each position ot^^ 
reached dh the best alternative. This process also helped fine-tune the test loop 
schematic. Both standard procedural approaches and individual SME heuristics 
were sought to develop the diagnostic tree^ 

Establishing the test loop schematic and diagnostic tree facilitated tevel 
I rule base develbprhent. Using an establish-fefihe approach, jhe rules 
incorporated a starting point general to all troubleshooting problems, and a 
diagnostic tree of ambiguity g^^ resulting from test outcomes. The diagnostic 
tree consists of levels of refinement and nodes associated with each leveh The 
nodes in the tree are either virtual Crepreseriting a set of actual nodes or test 
station corripbrierits) or actual (representing a specific test station component), A 
part of the diaghostic trTO developed for the stimulus side of the 6883 tes^t station 
is shown in Figure 15. _ The complete diagnostic tree was based on the structure 
and function of the ATEy the ease of testing, the cost of testing, and the 
likelihood of cbmpdheht failure. 
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Side 



Stimulus Source 
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Stimulus Logic 
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Stimulus Assembly 



Stimulus Assembly 
Input from DA TAG 



Partial Structural Representatibri bf the F-1 11 6883 Converter/Flight 
Control Systems test Statibh^ Shbwing Relationship of Actual (*) and 
Virtual Nodes. 



Level II uevelopment 

Level 11 of the knowledge base develbpment cbhsisted bf rules that 
directed fault isolation within the FCS Adapter. Unlike Level I, rules at this level 
were specific to the FCS drawer arid were nbt designed to apply to additional 
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TRUsi development of this^ rule base was relatively straightforward and can be 

described best a five-stage co^pera^dve process between knowledge engineers 
and SfcEs, First^ a hierarchiCTl structural representation of the FCS was 
constructed, based on^ the schematics,^ reference design^^^ 
breakdowns provided in the technical orders. Fi^gure 16 shows part of this 
structural breakdown for the FGS. Second, using the functional description and 
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Figure 16. Partial Struitural Representation of :he FGS Adapter, Including 
Reference Designations^ 



block diagrams, a fuhctibrial representation was developed; For the FGS, three 
basic functions were identified and are shown in Figure 17. The third stage 
required reconciling these two representations into a single system model which 
related structural components in a functional hierarchy. Although this approach 
was generally consistent with the reference designations, sbrrie components (e.g., 

ehassis-mbunted paftsj were redefined to facilitate this^ process. Fourth, a 

troubleshooting tree was constructed from the system model using the following 
basic guidelines in addition to the basic research assumptions: 

At the point where the FGS troubleshooting is initiated, the fault 
has been unambiguously isolated to the FGS. 

2; Each step in the troubleshooting process distinguishes at least one 
alternative in the next level of. the system hierarchy; 

3; The order in whicJi tests are prescribed is dictated by factors that 
include cost of test, likelihood of failure, and ease of repair. 
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Figure 18 . Partial Test Tree for Level li of the Maintainef 's A^ssociate Kriowledge 
Base. Circles are used to identify specification-based decisions that 
require no user input. 
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For the current Maintainer's Associate prqtoty DRi devekjped a totaj 
of 52 interaction frames, 130 probable cause rules^ and 130 evidence rdlesi Of 
these rules, approximately 60% were associated with the generic Level I of the 
knowledge base, and the remaining 40% were, associated with tevel !!• If it js 
assumed that the FGS drawer is fairly typical of all 6883 T 
expansion of the prototype problem domain to include additionaj components can 
be estimated at approximately 125 rules (total probable cause and evidence) for 
each new TRUi 

More important than the number of rules, however, are the results of the 
knowledge engineering process in terms of the nature of the rule bas^^ its 
structure^ That Js, in^ what ways did the outcomes of ^his process support or 
disclaim conventional expectations re^garding sevei^a^^ the knov^iedge 

engineering task^ includin^g the JO experts,^ the representation 

of troubleshooting test loops and strategies,^ and the types of rules that result? 
Admittedly, the Maintainer's Assqcjate rule base is lij<ely to represe^^ only a 
subset of the actusd trouWeshoo ting processes^ t^^ 

experienced technicians^ because SMEs often ernploy nnore information^ than they 
report or are even aware of usmg, Hov^ev^er, the c^nsjs^ 

obtained from the four SMEs contributing to the data base suggests that this 
effort has accurately reflected trouBjeshootinJ in a^vionics maintenance. For this 
reason^ ^_J^'^^§^^__o^_ g^Jl^^rfL^'PPM^^tio the present 

project are addressed. These include approaches to device modeling, the use of 
heuristics, and the selection and role of SMEs. 



Device Modeling 

It is generally assumed that an ejy^erienced^echnicjan's '<Q9^J'^_^g5L 

given troubleshooting situation takes the form of a, mental model of the device 
under testi Th^se models are not isomorphic representations of device 
topography; but rather, t hex are aDmjsqse^ ^ J^y^b?'" of interrelated and 
overlapping structures that are often hierarchical in nature. As is evident from 
the preceding descriptions of tevel I and II development, the prototype 
Maijitainer's Assodate rule base shares many of these characteristics. For 
exam pje,^ rules are hi erar^^^ both structural and 

^y^^tional^ relationships, and are not entirely parsimonious in terms of the devices 
represented. \ 

How^ever, there were several aspects of the experts' mental models that 

were not easily captured by t^ knowledge en gineerinjg procei^^^ and Rule-Kit 
architecture. These ambiguities j>res^^^ special problems tfa the knowlege 
engineering task and are ^_riefly described.^ Fi^'st,^ becaise a, single 
representational format was imposed, the resujtin^ rule base was /imited to^ only 
the most critic^ in terdependen^Jes^ Although the system aUo^ed structural, 

operational, and functional information to coexist in J:ne rule base, a 

comprehensive representation of all three was not practical. ' 
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Second, it was sometimes difficult to restfict the SMEs to the 
assurriptibns of the prototype and^tjie Jimite of the selected problem space. For 
example, since relays are a common source of malfuhctidh, SMEs sometimes 
skipped directly to th^a^t specific 1^^^^ even though ah intermediate 

level, such as a component assembly, was more appropriate in view of the 
Maihtairie^'s Associate design and the technician's task. A third problem, related 
to the previous two, was that arbitrary limitations on the system model did hot 
alwayp correspond with the meaningful levels used by SMEs. Therefore, flexible 
crj^teria were employed so that the resulting rule base was comprised of 
meahihgf ul units at various levels of structural detail. Fourth^ it is possible fbr_an 
overlap of functional, structural, and operational informatidh to occur. For 
^xample, fuses located on the TRUs of the ATE can be both indicators of faults 
(symptoms) and faults themselves. That is, a blown fuse implies a functional 
irregularity ih the test station and until it is replaced prevents normal Speratibh 
of the^test station. In addition^ a blown fuse may imply a fault in the TRU with 
which it is associated or in ar other TRU which is sending a (faulty) sighal to the 
TRU housing the fuse. Finally, it was found that the boundaries between device 
regioris or cbmpbne^^^^^ the Rule-Kit formalism. 

Cabjihg ahd bther connections are common sources of problems ih the 6883 test 
statibh, yet malfunctions of this sort lie in the interface betweeh regions, and, 
therefpre, ^cannot be attributed readily to a single compoheht. For example, lack 
of a firm ihterface between two pins can result in a test failure^ eveh thbugh 
none of the pins is broken or in need of replacement. The space between them^ if 
it is idehtifiable as an element, is at fault. 

Ohe solutibh to this interface problem is to consider each to be a distihct 
com pbheht that, when operating properly, is completely passive with respect to 
the sighal beihg propagated. However, carried to extremes, ihcorpbratibh of rules 
to address these interface ''regions" could cripple the system. A secohd sblutibn 
would be to arbitrarily assign interfaces a^ input or output elements of specific 
TR-Us br components, rather than as distinct componehts. Although certaihly 
feasible, the successful use of this solution requires extensive reliance bh SMEs to 
cbnsistently assign individual ij^^^^^ In the present effort this issue bf 

handling faults in interface areas did not jsrove to be a seribUs cbhcern. Most 
ihterface^ problems occur in the_attachment of the LRU adapter and LRU tb the 
test statibh, and these problems would be identified and corrected durihg the 
substitutiph of the shop standard tRU. Secondly, trbubleshbbtirig requires the 
removal of cables and other connections to perform signal measUremehts. Durihg 
this process, faults related to poor Interface connections would be purpbselyJor 
incidentally identified and corrected. Finally, in the opinions of the project SMEs, 
the likelihood bf an interface connection fault within the ATE is small. Ihterhal 
cbrri pbheht interconnects are rarely If ever manipulated, and that limits the 
bpportuhity for misalignment of pins. 



Heuristics 

particular type of rule often prominent in the trbubleshbbtihg 
literature is the heuristic or ruje-of- thumb. Heuristics are cbhsisteht with the 
technician's general troubleshooting approach and dictate jumpihg ahead tb a 
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likely soldtibn^ rather than systematically con^slderlnj; all alternatives. Sometimes 
the likely solution Is incorrect^ of course, and the_ technician m^^ backtrack and 
try a different alternative^ It was anticipated that heuristics would play a 
significant role in the developrrient of the Maintalner*s Associate ru^ 
l>ecagse experts often report using them to Identify malfunctions. As a result^ 
iumping ^d backtracking capabilities were ir^corporated into the Malntainer*s 
Associate system to accommodate rules of this type (see Chapter 3). 

However^ the SMEs consulted in this project made almost no use of 
heuristics even though they were encouraged to do so. A number of explemations 
for this iniexpected finding were considered. One possibility was that, since the 
primary SMEs were instructors rather tjian field technidsms, their approach to 
troubleshooting was essentially pedagogical, and thus sytematiCj rather than 
pragmatic. Furthermore, since neither instructor SME had substantial field 
experience,^ they may simply have beCT ur aware of usef^^^ Constraints 
oh field maintenance such as tjme^ personnel, and operational readiness^ may also 
provide a possible efxplanatibn, because these factors are not evident in the 
school envirbhmehti Finally, it was thought that the particular problem domain 
selected for the prototype^ especially the FCS, might be atypical with respect to 
the use of heuristic rulesi To exaniine these ejyjlanatiqns, the Cannon Air Force 
Base SMEs were consuitedi They could additional heuristics for the 

rule base and noted ttet heuristic^ were typically device-specific or aircraft- 
specific rather thrih appJicable at a general^systems level. That isj particular test 
stations or aircraft have Jdio^syn^cratlc malfunctions that recur under certain 
conditions; and in these specific situations, heuristics are particularly effective. 

- ^ hot her type of heuristic tfet Is ofiep dted in the troubleshooting 

literature involves the use of patterns of infqrnt?5tion (or symptoms) by 
technician. In this J<h6wl edge engineering effort^ ti:^se types of rules were also 
not in evidence. SMEs generally relied on a step wist ttf ■blish-refine approa^ by 
focusing bri a single piece bf da ta at any one time. :i >d patterns of information 
only develbped sequentially in the trou^bleshooting j^n^cesii the reasons for this 
are unclear, but it is likely that the perceived^ f nr nat of the Maintainer's 
Associate rule base, the digital nature ol the automat'c test equi vrnent, and the 
assumptions regarding the types bf allowable malfunction ^ all player* a role. 



Subject-Matter Expert Selection and Use 

As .previbusly nbted^ ^^^f^ t'l'^i^inS and^ -nsdntenanre j^;- /p environments 
provided SMEs fbr this wbrk. The tra'ming instnictcrs had ii:*;t??n5ive experience 
with the 6883 test station^ but limited field repair expei since. 1 hus, the 
experience bf the instructbr SMEs was aliped with an operc tion.U rather than a 
repiair/troubleshbbting context. On the other hand, the SivISs frcm Cannon AFB 
had substantial field experien^M OTd brought the opposite perspective— less 
emphasis bn representatibnal elements which m important to the novice 

technician and greater emphasis on the components particularly useful to 
trbublesHbbting. This difference had subtle J m pacts on the knowledge engineering 
process ir\ that the develbpment of the test loop schematic required substantial 
review^ even extensive mbdificatioh^ to reconcile operational representativeness 
with mairitenance/repair mbdeling. 
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The potential for disc^^ between the perspectives or. mental 

models of instructors and field technicians raised the knowledge engineering 
issues of SWE reljabiH Although the knowledge engineer can seek 

a^ consensus among SM^ points, lack of persdhal experience with 

either instruction or majnte^^ can limit the knowledge engineer's ability to 
resolve these larger questions. Once the model of the test loop arid the associated 
troubleshooting strategy had been modified, all four SMEs agreed that it was an 
accurate representation. This irnplied tha,t Air Training Command's 
troubleshooting training reflects field performance needs. However, some 
disagreement occurred in the area of domain ' .lowledge. Instructor SWEs were 
more familiar with all the areas of the test station; the field technicians knew 
more about specific arenas component failure rates, which were impbrtarit to 
developing the troubleshooting approach. 

For a particular deve^ effort, the decision to use multiple experts 

versus a single expert is most likely to be a practical one. In this project, 
multiple experts were used because of their availability and because the 
combination of instructor and fie^id ma^^ expertise was considered optimal 

in development of the troubleshoo and rule base. In practice, the 

selection of one or more experts will depend on the availability of a single 
recognized domain expert whose Information is considered reliable arid 
cbrriprehensive, the preference of the knowledge engineer for varied perspectives, 
or the ability of the knowledge engineer to establish multiple wofkirig 
relationships. The experience of this effort indicated that everi with multiple 
experts, each beco^mes, in 5 se^ expert because each exhibits expertise 

in a particular area or subset of the domain. Conflicts that arose among the SMEs 
were resolved through discussion and consensus. 



Although the approach to the Maintainer's Associate protdtypty 
knowledge engineering task was separated into two levels to address two different 
aspects of the rule base (the generic test Icup rule set and the TRU-spedfic rule 
set), many of the same elements were invovved in the development of the rule 
base for each Jevel. A j-n ^lemat'c, representative of the eqUipn^erit arid derived 
fiom a synthesis of exists schem atics and through discussion arid review with 
SMEs, was developed on v7;hic:\ to build a troubleshootirig strategy. The 
troubleshooting strategy refined^ again through discussidri with SMEs, and the 
rules which reflected the str^itegy ^'i^re developed, these ruleo we^e reviewed 
several times, especially in establishing test costs and sequences, ixnd resulted iri 
the final combined rule set. 

In the process, it a- discov : that limi prions of the expert system 
architecture used Jn the pr-st.; ^v-^>- ?e: Mlted i * constraints on the ability to 
trarislate the SME system^ modei v-> f . rule baj>a. In addition, houristics, 
frequently^iven emphasis in -x;;c^; r i^ v-t--; >erature, inay not be suitafcJ^ to all 
systenis. Their main importance ma. ' i %:em^ modeling specific aiiOraft or 
test stations, where modeling of t.,^v:,yn-ra: > r t the equipment would facilitate 
troubleshooting.. 
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Finally, the use of multiple experts did hot preseht sightficant problems 

regarding system modeling or _ trbubleshobtihg strategy; The opportunity to 
incorporate both instructor and field technician perspectives was very valaabie; 
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CHAPTERS: SYSTEM DEMONSTRATION 



A principal goal in this effort was to dembristrate ihe operations of the 
prototype Maintainer's Associate for a variety of Air Force, industry^ arid 
research audiences. These demonstrations were designed to meet three project 
objectives: (a) dissemination of results, (b) validatidh of the prototype^ and (e) 
formation of future R&D guidelines. The reactions of current avionics 
technicians were of particular interest because this iriformatibh provided the basis 
for _ system validati^on OTii the prototype as a job 

performance ajd and trainer. As c: complement to this perspective^ the reactions 
of maintenance of ficers allowed project staff to assess the resF>6nse to integrated 
diagnbstie equipment that may be expected for future weapk)n and support 
systems. The results of these technical demdnstratibhs are presented in this 
chapter. 



Demonstration to A vionics techhiciaris 



Approach 

Por technical personnel, a scripted dembristratibn was developed based 
on the_typicd troubleshooting scenario which was introduced in Chapter 3. This 
sceriarib began when a failurr was indicated at test_riurriber 30J982 in the 
automatic test sequence for thv Feel and Trim LRU. By substituting a known 
operational Feel and Trim into the test loop and repeating the test sequerLce^ the 
problem was isolated to the test station. Subsequent manual troubleshooting 
proceeded under the guidance of the Maintaiher's Assbciate system. The display 
shown in Figure 19, for example,^ requested that the technician cheek test points 
14 and 15 on the front panel of the FCS. Eventually, a malf unction in a power 
supply circuit of the Fes adapter was identified arid apprbpriate repair action was 
indicated. This interactive sequence of Mairitairier's Associate displays and 
keypad responses was selected to exhibit the full range j)f system features within 
the context of a fairly typical test statibh prbblem. The complete series of 18 
demonstralion displays required approximated 10 riiiriutes to present. FoUowirig 
the scripted demonstration^ technicians were ericburaged tb try but the 
Maintainer's Associate by entering one test huriiber frbrii a list of 60 that the 
system was capable of troubleshootings technicians would bpen their LRU 
technieai orders to the test number in question arid corii pare the MaintainePs 
Associate diagnostic strategy and specific requests for measurements to their 
own. Bemonstrations were conducted with rib mbre than three techhiciaris per 
system to ensure clear visibility. Each sessibri was prefaced by brief remarks 
about the purpose of the system, allowed time for questions and answers, and 
included a period for hands-on trybut bf the MaintainePs Associate. 

Two different technical grbups were selected to participate in the 

demonstration sessions. The first consisted bf six avionics instructors from the 
Air Training eommand at Lowry AFB, Cbloradb. These technicians averaged 
more than 6 years of avionics maintehance experierice, and all but one were rated 
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at the 7-level skill dlassificatidn. The second group was comprised of 10 avionics 
technicians from the intermediate^level F-111 malnfenance shop at Cannon AFB^ 
.New Mexico, who averaged approximately 2.^ years of experience. A broad range 




Iriteractloh Frame for Checldng Test Points 1* and 15 in the FCS, 



Results and Discussion 

System critiques . Critique forms were self-administered by all avionics 
personnel who participated in a demonstration session with the Maintainer^s 
Associate. Personnel were asked to assign ratings to each of 12 system 
performance factors, using a 5-point Likert scale that ranged from "very poor" (1) 
to "excellent" (5). Space was also provided for general comments and comments 
about those features they liked most and liked least. A copy of the critique form 
used is provided in Appendix B. 



Table 2 shows the mean ratings for each performance factor as a 
function of each group and overall. In general, the responses were 
overwhelmingly positive, with both instructors and field technicians giving the 
system a mean overall rating of 4.6. The lowest combined mean rating concerned 
the range of user options (3.8) and the highest combined mean ratings were given 
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for ease of use (1.8) and usefulness for training ($,8), Surjjrisirigly^ although the 
system was desjgned as an on-the-job troubleshooting aid^ both groups rated it 
slightly higher on its usefulness for training than oh its usefulness for job aiding.. 
The rnost noticeaJ>ie^^d^^^^ ratings of the two groups (field 

technicians vs; instructors) were revealed in their assessment of the display 
quality [ 3.8 vs; 5.0; t(l«) = 3.28, £<. 001] , the helpfulness of explanations [ W6 vs. 
3.5; t(I2) = ^.16^^ <.01 ] , and the usefulness for job aiding [*.7 vs. 3.8; t(I^) := 
2.59, £ < . 05 ]. These 1^^^ suggest that instructors may not be in 

the best position to judge what is perceived as helpful and useful by less 
experienced technicians in the field. 

Table 2. Mean Ratings^ of System Performance from Demonstration Critiques 





Field technicians 


Instructors 


Overall 


Performance Factor 


(n = 10) 


(n = 6) 


(n = 16) 


Ease of Use 


*.7 


9.8 


9.8 


Speed of Gperatlon 


*.l 


9.5 


9;3 


TrbublesHooting Accuracy 




9.4 




TrdtiblesHdoting Strategy 
TrbublesHooting Efficiency 


%.7 
U.3 


9.5 
9.0 


9;6 
9;2 


Range of User Options 


tf.O 


3.3 


3.8 


Display Quality 


3.8 


5.Q 


9.3 


Helpfulness of Explanations 


tf.6 


3.5 


9.3 


Hardware Packaging 


%.2 


n.l 


9.9 


Usefulness for 3ob Mding 


%.7 


3.8 


k.k 


Usefulness for Training 


5.0 


9.3 


9.8 


Overall Rating 


ft. 6 


9.6 


9.6 


^Scale; 1 - Very Poor, 5 


= Excellent 







Additjonai comments were also favorable and fpcused largely oh the 
training potential of the prototype. A number of the technicians discussed the 
need to provide more depth to the system in terms of the malfunctions covered, 
explanations provide^, and technical references. The features that they 
reportedly liked th^^ the Maintaiher's Associate included its simplicity 

and training capabiiity, its compact size, its logical (test-loop-based) approach, 
and Its ability to save time by avoiding the use of technical orders. Those 
elements liked the least included the readability of the display (too crowded) and 
the lack of depth in the range of problems arid scope of explanations in the 
current prototype. Several instructors also expressecl the concern that novice 
technicians nii^ht become dependent on the device without developing aDDroDfiate 
troubleshooting skills. r o k 
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Fbrmatiye irLter views . Individual interviews were also cbhdUcted with 
five of the field tecHhiciahs to assess in more detail their reactions to the 
Maihtaiher's Associate, arid their suggestions for future work. Each interview 
lasted approxinlately 20 rtiiriutes arid cbrisisted of bperi-erided questibris about 
general system operation and each of the system interface features, the 
iriteryiew guide used to sUppbrt this data cbllectiori effbrt is prbvided iri Apperidix 
B. The results of these forniative iriterviews were gerierally cbrisisterit with the 
critique results discussed previously. Specifically^ the field techriiciaris reported 
that: 

1. the message lerigth arid level of explariatibri prbvided by the HOW 
feature were appropriate^ but the additibri bf technical brders refererices niight be 
useful; 

2. the graphics were quite helpful withiri the iriteractibri frarties^ 
especially the test Ibbp diagrani^ but the screen appeared a bit crowded at tinies; 

3. the MARK & RETURN feature cbuld be iniprbved by allbwirig ^he 
user to begin anywhere iri the trbubleshbotirig prbcess; 

^. the WHERE-TO feature was useful but might be mbresb if the 
irifbrmatibri was pjresented iri a test tree graphic fbrmat; arid 

5. the ffi^'HERE-FROM feature was critical for trairiirig purposes a^ 
might be improved by the additibri bf cariried explariatibris arid/br test tree 
diagrams. 

Although comments were bverwhelriiirigly favorable^ additibrial 
suggestibris iricluded expandirig the problem space to bther TRU arid everi LRU 
malfurictioriSj^ addirij a hard-copy capabilityj arid providing more detailed 
irifbrmatibri for riovice techriiciaris. Techriiciari's cbriiriierits cbiricided in large 
part with the expectations bf the prbject staff arid were typically a reflection bf 
the established scbpe bf the prbtbty pe effbrt^ rather thari systerii desigri 
limitatibris br bperatibrial shbrtcbniirigs. Suggestibris fbr future system 
develbpmerit are corisidered riibre f ully iri Chapter 6. 



Denibristratibri tb Deputy Chief bf Mairiteriarice 



Apprbach 

_PRI preserited a fbr mal_ briefing arid scripted deriiblristratibn_tb the 
Deputy Chief bf Mairiteriarice_(DCMj arid members bf his staffs 27th Tactical 
Fighter Wirig^ Cariribri AFB. The briefirig explairied the backgrburid^ bbjectives^ 
arid techriical approach of the prbtbtyf>e- The scrij^ted demoristratibri was the 
same brie preserited tb avibriics techriiciaris; each denibristratibri took about 20 
miriutes fbllbwed by over an hbur bf discussibfu The pUrpbse bf these sessibris was 
tb validate the Mairitairier's Associate cbncept^ ribt with respect tb the accuracy 
of its techriical detail^ but with respect tb its feasibility f rbrri ari brgariizatibrial 
arid mariagemerit perspective. 
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Results arid Discasskm 



. J°"°«'»"g the briefing and demonstration can be 

fit^'??"^ ' r ''^^ First, the Maintainer's Associate, or devices 

Like It, were famiimr to maintenance officers; and the likelihood that the Air 
^S^n Jn Ir^"',^?."^ ^""^1°^ method of job aiding was acknowledged and 
accepted. The DCM took a realistic view of the Maintainer's Astbciate, 

£a?Sfon^S^'«^- f ^ "^^ technology, there is a need for system test 
evaluation, and continual imprbvemerit. 

, Second, the maintenarice officers Were cori^erned with being deperiderit 

on yet another comjjuter system. Computer-based systems have a reputa^ri for 
^^f^^'y ^ The .fffcers rtcommended t^t S! t^^ 

backup system be provided as part of a devic- '^^^ployment. 

^- ^ group^^ w about the potential foi mental 

dependence on the job aid. Since the WaiTitainer's Associate is capable of solving 
trouhleshooting problems, the officer, -^.re worried that avicrSs t °chriid£f 
J^^^^^^ "ot develop or exercise their own diagnostic 

competency. The formal brief irigb.-llined how this^ issue was explicitly addressed 

of skill multiplier features, iricluding 
maintenance training simulatibri. The prototype was purposefully coris'trUctlS riot 
only to avoid mental de pen derice, but to actively support skill acqulsitlbri. It is 
valSri " 'n'i officers' concerns with^'dependence Teinfo^ced the 

nSn for^ • .P^'P^'^ °* multiplier features-tb satisfy the 

need for trained technical personnel. ^ / «.■«: 



Cbriclusion 

r^^t\..\^^^ k"^- ;^*=^"|*=^\ audience for the system demonstratibn was 
relatively small, technicians clearly validated the approach as implemerited in the 
current prototype. Both classroom iristructors and field personnel were favorably 
A^soru'f^ with the performarice^ and training potential of the Mairitainer's 
Associate. Their suggestions fbr future R&D focused on realizing that pbteritial 
and expanding the problem domairi. The management audience wis not resistant 
to ^he associate concept as the irievUable solution to existing mairiteriance 
fhf^ S^L^i however, concerned with how 

b^^hl?m^ ^ F — be institutibrialized, especially with respect tb the pbteritial 
problems of physical and mental deperiderice. ai 
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CHAPTER 6. CONCLUSIONS AND DIRECTIONS FOR FUTURE WORK 

H^^}^r.J^^^'^^°^ ^'^^^^^^ ^ nurtibeiLbf pbteritial aFeis for future 

first, to summarize the accomplishments of the current work and. sTcond S 
make recommendations for future R&D in iight of the anticipSed near-term 
trends ,n weapon support systems. These objectives are addrSsed af they refatS 
to speafic topicai areas of R&B in the discussion that foiiows. 

Skill Mditipiier 

i ^ Next-generation weapon systemi will continue the lone-standine trend 
toward greater complexity. By virtue of this, increased diagnS^c siph sticltlCn 
will be required just to keep even with current mmntenance profidSc^^^ 
increased diagnostic sophistication, however, will not include mine use of 
automatic test equipment in intermediate-level shops. On tS contrary thl a1^ 
Force will move toward reduced dependence on the avionicfinlrme^ite shoi 
Je^ilbTe^'Ruf °^ strate^yrlnau^S ^ 

.^^Mn^/^on '^\e^hrm%--Smd^ 

production, and fault-tolerant design. Human involvement iri weapon "sys-m 
ASocia5''%his ••^^^"' P'^^-SIy supported by devices like the laintSn^:^. 
Associate. Thus, the weapon system, its automatic support systems and the 

in tSLy""y~' "^^'"^ sophisticated dia^gnlsti^cafly'th^' 

its diagnIsL'*^!^L'2?.'?*'°"' of the prototype Maintaiher's Associate established 
Its diagnostic competence and appropriateness for the field environment 
Technicians perce ved it as a good job aid and training device, the two p^rciil* 
°* enables nivice techSdans ?o Si"?; 



^gnostic^roblems beyond their own level of com^et^nS^ar^d tS^^fSn'^ 
to solve future problems on their own. 



exercises P^^t^skiilsacqm^ is through problem-solving 

exercises. For maintenance technicians, these pfOblem-sblvinK exercises 
generally take the form Of troubleshooting simulations. Given the fact that the 
Maintainer's Associate has diagnostic competency in the expert system and Jiven 
that it IS possible to access the reasoning or inference rSechanSm BeWnf this 
competency, it provides the basis for construction of a trOuwSSgloach ^n 

(r|fer to Chapters 1 and 2), this coach is simply aninversion 
of the expert system. The technician poses the questions as to where and what to 

^^^^V '^'^^ ^'"^^'^ "^'^ the (simulated) finding The "coaS 
•f^K \ ^ to anticipate each of its own processing steps in advance- an? 

If the student's next step differs from the coach's, it intervenerimmediSlfv with 
corrective action. The inversion, however, can, 4nd shSTS d^^ Vfrnn ^ 
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By designing the coach to the extent that the technician is forced to emulate the 
actions of the expert system's inference procedure^ the technician learns a 
successful gen erai diagnostic strategy (i,e,^ ^V^l^ 9^ ^^^^^j^^f^" 

refine) and the problem-specific details (iie,^ the rule base) on which it operates; 

Future work should focus oh implementing this coach and coupling it 
with a curriculurri-sequehcihg module and a student model to yield a complete 
Intelligent tutorial systerhi The curriculum-sequehcihg module >v6uld be able to 
determine what simulated fault (and what part of the troubleshooting task for 
that fault; out of all possible faults would best serve to advance the skill level of 
the tcchivcian as measured against on-the-job training objectives; The student 
model v^.jld be a detailed record of the technician's successes and failures with 
e^^'^h simulation exercisei 

A key outcome of this work would be demonstrating that the same 
knowledge base useful in job aiding is useful in traihihgi Were this true, it would 
pave the way to the successful integration of job aiding and training and present a 
revolutionary hew way to develop the skills in a cadre of techhiciahs. More 
generally, the succc 5 "^'il d<*vel6pment and evaluation of such a troubjeshooting 
coach would suppleroent what is khowr about building intelligent tutoring systems. 



Skill Integrator 



The design goal for future weapon systerhs will hot only change, but the 
process by which this goal is achieved will be hewi First, fault detection and fault 
isolation concerns will be addressed as a system and the various diagnostic 
technologies — Br-^^ ATE^ job aids^ MIS — will not be developed independently, but 
as ah integrated wholci Cornptiter-aided design, engineering, and rriahufacturing 
have already had, and will continue to have, a large impact oh the design processi 
This computer-aided support capability establishes a closed information loop from 
design to support, and back again, throughout which the device model serves as 
the basis of corhmuhicatioh, coordination, and integration arhong the phases of a 
weapon systerhi This closed-loop approach reinforces the idea that systerh 
maturation never ends. Because of their important role in serving as a source of 
information on weapon and support system perfofmahce^ trained technical 
personnel will always participate in maintehahcei The skill integrator interface 
of the Maintainer's Associate was designed to support this rolci 

The skill integrator works cooperatively with skilled technicians to solve 
problems arid capture the human technician's diagnostic insights as they arise. 
The fbundatioh for this feature is the extensible, modularized rule base of the 
Rule-Kit expert systeriii The expert system architecture is capable of 
interpreting, in a uniform representation^ rules of inference derived from a 
descriptiori of system structure, as well as informal^ experiential heuristicsi 
Although this basis for adding human diagnostic insights to the rule base exists, a 
suitable debriefing interface was not implemented in the MaihtaihePs Associate 
prototypes 
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Future^work to be done on tWs debriefing interface offers a vaFiety of 
\^*«f^',^yehts could initiate the debriefinj, ranging from user initiative 

K.^r K ' ^f'u""^ *° ^" <=^se, the architecture ind 

knowledge base of the expert system could provide context, semantics, and even 
tL" "^^'^ *° carry on a discourse with the user regarding a new diagnostic 
rule The problems of natural language are greatly simplified When the context is 
constrained and when semantics and syntax are specified. The existing 
architecture and rule base makes the notion of talking to a computer about an 
insight a far more jikely possibility. Successful implementation of this debriefing 
interface would yield results of general interest to the field of natural languagl 
processing. 5"°5'= 

A second asjject of the maintainer's associate skill integrator interface is 

cooperative human-computer problem-solving. Very little basic theoretical work 
has been conduc_ted in this area. The problem-solving strengths of humans and 
computers have b^en contrasted in a descriptive way, but no prescriptive design 
for combining these has been approached. The broWse, jump-ahead, and 
debriefing features of the Maintainer's Associate are only a beginning^n what may 
become an increasingly important field. Basic R&D in this area could be focused 
domai'n^ problem-solving to troubleshooting in the electronics equipment 

- ^ particularly important that an associate be able to work with both 

S?v^?nTiTc^^ T^"'mI^'^ ^^ys. The novice may 

reiy on the syster.i for all diagnostic reasoning and data-gathering. The exoert 

^nTTT "^^^ ^'^^ observations and%reliminary ryonfng 

have probably been done iridependently. coauimig 

i * RETURN feature on the Maintainer's Associate 

toSnriloh^ partial solution to tWs concern. This allows the technician 

to explore where Jhe system would go, diagnostically, given different answers to 
ts questions.^ The technician, however, is still unable to mako assertions 
independent of whether the assertion is an acceptable answer to the question the 

t^k??RrSt^?°'^' K ^ ''^^^^^^^^^ technicians should be able to 

take the initiative by making^uch assertions, both of evidence observed and of 
proteble causes suspected. These assertions should also be able to be made 
tn rntt *^f^^°"*^^\of the expert syste^^^ state (i.e., they should not have 

to match the assertions the expert system is currently processing); and once the 
bei^ng made!"^' assertions, they should in^pact the flow of inferences 

iK.i * r* ^ ^^^""^ features represent only one of many po-sible design strategies 
^hat facilitate cooperative numan-computer problem-solving. It is an approt h 
that mi tigates_ against the inevitable frustration experts Would feel in dealing wi^h 
fr/h^t?T "-'^ices^ The Maintainer's Associate expert system 

architecture provides a foundation for further drv-ibpment of the 
mixed-initiative ajjproach to troubleshooting. 

in thu ^g"f^^^P^*=*^°* the.maintm concept that was not explored 

m this effort was the ability to prebrief technicians. A prebnefipg interface 
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would integrate corporate- wide skills by prc »*Vdirig tecHhiciahs access to the MIS 
records for all past repairs of the system under test. The development of MIS 
systems has been accomplished for many specific weapon systems. In the future^ 
these systems should capitalize on a design to close the irifbrrhatiori loop based on 
the device model, as discussed in Chapter 1. MIS Systems would also benefit f rbrti 
the notion of episodic memory arid the use of AI techniques in this area (see 
Kolodrier, 1983) to develop the database of maintenance events. This could be 
useful in characterizing arid categorizing mainteriarice events; iri recdghizihg rieW| 
riovel, or aberrant events; and in facilitating the returri fldw of inforrriatidh back 
frorii support activities to desigri activities^ where rectifying design changes may 
be made. 



For the prebriefirig iriterf ace^ future work should also be directed at 
rouridirig out the complement of four irifbrmatibri resburces which the Maihtaiher's 
Associate incorporates. This wbuld iriclude the iricbrpbratibri bf ah iriteractiye 
gateway to the weapon system's riiairiteriarice irifbrmatibri system arid tb its 
supporting tecHriical documentation, priricipally schematics^ illustrated parts 
breakdowris, and removal, installation, calibratibri, aligririierit^ safety, arid other 
information. 



Kribwledge Acquisitibri 



Hybrid Diagribsis 

Demonstrating an efficierit kribwledge acquisitibri strategy capable bf 
iritegratirig both specification- and symptbrii-based kribwledge was a project 
objective that was achieved largely by represeritirig all rules iri the 
establish-refine formalism. Although no autbmiated tools were used tb develop 
specification-based diagriostic rules, the knowledge erigineers, iri corijUrictibri with 
theSMEs, used a specification-based process. This process analyzed the structure 
(topology, dependency) of the systerii under test, derived a dlagribstic strategy 
(test tree) from this, and converted the strategy tb rules in Rule-Kifs syritax. 
Alt*^6ugh this process was conducted using the techriiciari's merital mbdel^ its basis 
was system structure, not empirical syriiptbrri-fault assbciatibris. (This process 
and. its results are described in more detail iri Chapter 

It must be recognized that however well a dlagribstic system may be 
operatibnalized usirig orily specif ieatibri-based kribwledge^ sbriie faults will go 
undetected because of iricbriipleteness or simplificatibri in the device model. 
Symptom-based rules fill the gap iri the kribwledge base when the manifest 
symptom-failure assbciatibris for previbusly uridiagribsable faults are deterrhiried. 
This project showed that a hybrid diagnostic apprbach bffers a feasible methbd for 
res6h*ing this shbrtGoming bf specification-based diagnostics. The prbject staff 
found, however, that for this dlagribstic task, the great prepbriderarice bf rules 
were specificatibri- rather than symptbrii-based. Knbwledge engineers iriitiated 
their work cbriscibusly seeking but each type of irifererice^ but in only a very few 
cases was syriiptbrii-based knbwledge employed. Wheri fburid^ however^ it was 
pbssible to use symptbrri-based knbwledge tbjgether with the specification-based 
knbwledge in a unif brm representatibri. 
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Reconfigurable^Svstems 



A major contribution of this jjroject was the implementation of ah AI 

approach to troableshooting recorifigurable systems. The 6883 test station is 
really ^Qd different systems, depending on the tRU test number. The chahenge 
posed was how to troubleshoot a system of multiple states without developing 
multiple sets of rules. The solution was to conceive of the system at a level of 
at>straction at which the system's description remained invariant across states In 
the case of automatic test stations^ this abstraction was the test loop. (Other 
reconfigurable systems will have other invariant abstractions.) The system was 
abstractly conceived, and the diagnostic strategy was applied to the compbhents 
ol the abstraction. This was an instance of the hierarchical decomposition 
approach to diagnosis, where the hierarchical abstraction not only removed details 
of connection but also of state. 

. . ^... J^^^^^BPostic stra^^^ derived from the abstract system modei 
Identified where to test (in terms of topology or dependency), but not where to 
test _ physically. Although abstracted, the system itself remains a concrete 
physical object. Thus, there was the problem of how to make the diagnostic 
strategy's request to check at the input to an abstract region match the actual 
correspondin| physical location and expected signal value. This translation was 
accomplished t^hrough use of a prser, which compiled lists of physical locatibhs 
and expected measurement values by abstract region for each system state, the 
parser s principal source of knowledge about system state was the UUT's test 
program set; indeed^ the test program set is what defines the test station's state. 
J he diagnostic test strategy was written independently of state, and the 
instantiatoon of the abstraction into a physical location and expected value was 
accomplished by the parser. 



Authorir 

, There are three types of tools that can be of particular use to the 

knowledge engineer: (a) a rule base editor that operates in the context of a 
diagnostic consultation, (b) a verification routine that exercises all possible paths 
through the rule base and reports inconsistencies, and (c) a system-specificatibh- 
to-diagnostic-strategy converter. 

^ , °^i^<=*ive of this work was to demonstrate that knowledge base 

development could be accompilished by nonprogrammers. This was achieved at the 
outseg simply due to the nature of the expert system architecture selected. In 
Kule-Kit^control and data are separated. A uniform data format is interpreted in 
a uniform way by the expert system's inference engine. Once a knowledge 
engineer has interpreted a dia^nb^ in the context of the expert 

system s architectiTe and rule format, it is easy to develop the knowledge base. 
1 his project made -.-e task of the j<nowledge engineer even easier by providing the 
ability to enter or edit rules in the context of their execution with the glasi box 
editor. Thus, the glass bbx editor enables nonprogrammers to input and 
manipulate the knowledge base. This editor may be a good starting f^int for 
further work on the skill Integratbr interface. The graphics work station and the 
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validator-verifier were other system development tools which helped ccihplete 
and debag the knowledge base. Many routine procedural errors were detected by 
the validator-verifier and then removedi 

In this particular project, an aid for converting descriptions of structure 
into diagnostic strategies was_^ reviewed in Chapter 1, many 

such tools exis^ Since the knowledge engineers employed largely a specification- 
based approach, it is concluded that this stage of knowledge base development 
could be supported with a computer-based tooh The tool should permit flexible 
interaction with the knowledge engineer, both in inputting the device model and in 
interactively editing the tool's outputs 

Institutionalization 



Issues surrounding the organizational impact of a hew technology must 
be addressed far in advance of efforts to use the teehnologyi Before advanced 
development activities with the Maihtaiher's Associate can be undertaken, issues 
surroanding i^s institutionalizat^ to be exploredi^ 

issues that wens raised during the dejrion^stration^s at the end of this project need 
further examination: reliab How can the maintainer's 

associate system be made sufficiently reliable so that it performs reliably in ah 
operational environment? how can users be 

persuaded and convinced of this reliability so that they accept this new approach 
to technical job aiding and training; and what other organizational factors will 
enhance its introduction, acceptability, and utility? 

Advanced development efforts with associate systems will also require 
addressing systems ehgiheefihg issues. The overall maihtaiher's associate system, 
and its development and operation, would need to be explored in detail. Ideally, 
the system would comprise a worldwide information network, with links from the 
design engineers to the flight lihei Important questions immediately, arise with 
respect to this scertsrio: How will configuration control be maintained? How and 
how often will updates be managed? How will this information, tahtarhouht to the 
health, well-being^^d weakne^ our weapon systems, be secured? What will 
^^?PP^"_ L^ l^? system's^ satellites are down? How will the system work if a unit is 
deployed to a remote area? 

Systems engineering issues also extend to the classrbbrhi The integrated 
job aiding and training a maintainePs associate makes possible will undoubtedly 
have an impact oh the Air Force's training establishment, the Air Training 
Commandi Specifics of this impact need to be identified and varieties of 
responses need to be ahstlyzed so that changes can be made in anticipation of, and 
not in reaction to, the changing technology that will bring the maintainer's 
associate to fruition. 

Due to the Recess in^ 4^^^^P_P^PS__ the protoype, its successful 

deniqnsjration, and t^ and scenario of th^ ^^^^^Al}^ Js 

recommended that advanced development versions of the Maintainer's Associate 
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•'^ v«^°P«<* and fieid tested as soon as is practicable* This recommenStiwi is 
gO"f**«"* witfi the finding qf the National Acadeiny of Scdences eommittee on 
Fault Is^ation in Air Force Weapon and Support SystemSi The prototype, without 
any of ^the enhancements possible with exploratory development^ if scaled up to a 
realistic scope, could ^rove to be a useful and acceptable advance in weapon 
systems Mppqrt. The diyjloym^^^^^ such a maihtainer*s associate system for an 
Air^Force weapon system is an achievable near-term »>al. In preparation for this 
goal, serious studies of the benefits, costs, and risks of the maihtainer>s associate 
need to be undertaken. As a first step, this project has helped define the 
concept, reduce the risk involved, and identify the bmefits that might accrue. 
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APPENBIX A: TEeMNICAL DATA FOR PARSER DEVELOPMENT 



Technical orders 

manaai Figure 
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33D3-9-i03-2 


6-1 




8-2 




8-6 


33D7-15-2 


9-1 




9-10 




9-11 




9-1* 




9-15 




9-16 




9-18 


33D7-«2-i-i32 


*-l 








7-m 




7-15 




7-16 
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33 DAS -2 1-2 
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2-lb 




2-11 
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APPENDIX B: DATA eOLLECTIdN INSTRUMENTS 
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CRITIQUE 



Please take a few rtiiriutes to record your impressions of the prototype 
Maintainer's Associate (MA) that has just been demonstrated. Your commeTits are 
vital for evaluating the success 61 the project to date as well as for determimhg 
the direction this research might take in the future. Thank you] 

What is your current skill level? 



How many years experience do you have in electi briics maintenance? 
How many years experience do you have with 6883/73 test equipment? 
Did you actually use the MA? or just observe the MA? 



How would you rate the performance 61 the MA system on the following factors: 





Very Poor 








Excellent 


Ease of use? 


1 


2 


3 




5 


Speed of operation? 


i 


2 


3 


k 


5 


Trbubreshodtihg ac* aracy? 


1 


2 


3 




5 


Troubleshddting strategy? 


1 


2 


3 


k 


5 


Troubleshooting efficiency? 


i 


2 


3 


u 


5 


Range of user optic ris? 


1 


2 


3 


k 


5 


Dispilay quality? 


1 


2 


3 


it 


5 


Helpfulness of explanations? 


1 


2 


3 


k 


5 


Hardware packaging? 


i 


2 


3 




5 


Usefulness for job aiding? 


1 


2 


3 


k 


5 


Usefulness for training? 


i 


2 


3 




5 


Overall rating of perf drmarice? 


i 


2 


3 




3 



What did you like most about the MA system? 



What did you like least about the MA system? 



Any other comments? 
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INTERVIEW GUIDE 



Date: 
Time: 



Lbcatibh: 



Subject: 

Interviewer: 



1. Read the fbllbwihg iristructibris: 

"The purpose bf this trybUt session is to get your honest reactions 
tb the varibus features bf the Maintainer's Assoeiatei This is a 
prbtbtype system, and ybur comments are important for fature 
develbprheht effbrts. For the first part bf this session, I would Hke 
tb juide ybu thrbujgh a possible troubleshooting situation, 
cbllecting ybur reactibns (if any) at each step in the process. 
During the second part bf the session^ you will have an opportunity 
tb try but the MA system entirely bn your own;" 



2. Begin the demonstration part with the SME entering all choices. Try to get 
comments bn the following features as they seem appropriate. 

a. HOW 

Appropriate^ level of explanation for 3- and 5-leve! 
technicians? 



Message length OK? 



Need didditional details, references, or diagrams? 



b. INTERACTION FRAMES 
Are directibhs clear? 
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Does the system have the right 'associate' tone^ or too 
authoritative? 

Are the graphics useful? 

Are the alternatives clear arid appropriate? 

c. MARK & RETURN 

Is this feature useful? 

is it easy to Use? What would rriake it ' etter? 

d. WHERE XO 

Does this feature give ybli enough explar» re ;<o'r. 

Is it useful? 

What might make it better? Test loop diagrams? Binary test 
tree? 



e. WHERE FROM 

Is this feature important/useful? 

is the iriforrriation presented clesarly? 

What might mak;^ 5t better? 
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3. Allow the technician to try out the system for a short time alone. Record 
any comments or problems^ 



4. Question the technician about additional features and future jjlans for the 
system (e.g., possible notes file, simulation capability, canned why's). 



5. Question the technician about hardware features. 



a. Speed of response 



b. Overall size 



c. Display quality 



d. Inpi: . } 



6. Have the technician complete the standard eRITIQUE and attach to this 
form. 
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